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INTF ODUCTION 


The  selection  and  use  of  commands  is  one  of  the  most 
basic  skills  required  for  proficiency  in  a  software  system. 
Mastery  of  the  command  language,  however,  is  no  easy  task 
for  inexperienced  users.  Users  must  not  only  learn  the 
names  and  functions  of  commands,  but  must  also  learn  the 
situations  and  strategies  for  using  commands  in  a  task.  In 
this  dissertation,  online  aids  were  designed  and  evaluated 
to  help  novice  users  in  command  selection.  Specifically, 
the  research  assessed  the  feasibility  of  online 
command-selection  aids  for  information  retrieval. 

Previous  research  by  Elkerton  and  Williges  (1984a,  1985) 
revealed  large  differences  in  the  performance  and  strategies 
of  experienced  and  novi  e  subjects  in  a  file  search  task. 
The  results  of  these  studies  illustrated  that  experienced 
subj  -S  not  only  used  more  powerful  search  procedures,  but 
also  used  these  search  procedures  quickly  in  the  appropriate 
situation.  In  contrast,  novice  subjects  selected  a  wider 
variety  of  search  procedures  and  tended  to  use  search 
procedures  that  manipulated  the  file  search  display 
directly.  Thus,  novices  lacked  both  the  knowledge  and 
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strategies  of  file  search  that  more  experienced  users 
possessed. 

As  a  result  of  the  observed  difficulties  in  the  selection 
and  use  of  commands,  Elkerton  ^nd  Williges  (1984b) 
implemented  an  online  assistant  to  aid  novice  users  in  file 
search.  The  online  aid  was  based  on  the  selection  of  search 
commands  by  experienced  users.  Essentially,  a  frequency 
profile  of  the  search  commands  selected  by  experienced  users 
was  provided  automatically  to  novices.  The  results  of  this 
research  revealed  that  the  frequently  selected  search 
procedures  of  experienced  users  could  serve  as  a  training 
device,  but  were  severely  limited  in  assisting  novices  on 
current  information  retrieval  problems. 

Elkerton  and  Williges  (1984b)  explained  the  mixed 
effectiveness  of  the  file  search  aid  in  terms  of  the 
limitations  of  the  frequency-based  model  and  the  constrained 
dialogue  with  the  user.  The  frequency-based  model  offered 
only  limited  information  on  command  selection.  Frequency 
data  provided  no  sequence,  planning,  or  procedural 
assistance  to  the  novice  user.  Together  with  the  one-sided, 
computer-initiated  dialogue  for  offering  advice,  it  was 
remarkable  that  novices  could  use  the  command  frequency 
information  at  all. 
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How  can  the  knowledge  and  strategy  of  command  selection 
be  conveyed  naturally  to  inexperienced  users?  Two  issues 
are  important  in  answering  this  question.  First,  how  should 
command-selection  strategies  of  novice  and  experienced  users 
be  represented?  Second,  once  strategies  are  represented, 
how  should  the  more  skilled  strategies  of  experienced  users 
be  presented  to  novice  users.  The  goal  of  the  dissertation, 
therefore,  was  to  develop  and  evaluate  systematic  methods 
for  capturing  and  presenting  the  command- selection 
strategies  of  experienced  computer  users  to  novice  computer 


BACKGROUND 


The  study  of  online  aids  for  inexperienced  users  was 
motivated  by  research  on  intelligent,  human-computer 
systems,  research  on  supporting  the  learning  processes  of 
inexperienced  computer  users,  and  the  preliminary  research 
of  Elkerton  and  Williges  (1984a,  1984b,  1985)  on 
command-selection  aids  for  information  retrieval. 

Intelligent  Human-Computer  Systems 

The  advance  of  computing  technology  has,  in  the  last  20 
years,  introduced  the  capability  for  intelligent, 
human- computer  interfaces.  Computers  are  now  capable  of 
assisting,  instructing,  or  adapting  to  individual  users 
(Williges,  Williges,  and  Elkerton,  in  press)  and  can  no 
longer  be  assumed  to  be  passive  tools  for  human  use.  This 
development  in  computing  systems  has  been  largely  a  product 
of  artificial  intelligence  (Hillman,  1985)  and  until 
recently  has  not  been  addressed  by  human  factors 
specialists.  However,  the  need  for  human  factors  research 
to  guide  the  development  of  these  more  intelligent 
interfaces  is  of  great  importance. 
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Unfortunately,  there  is  little  human  factors  data 
available  for  research  and  design  of  intelligent 
human-computer  systems.  To  fill  this  void,  human  factors 
specialists  should  anticipate  the  technology  of  artificial 
intelligence  to  provide  the  necessary  data  to  construct 
effective  and  intelligent  human-computer  interfaces. 

Clearly,  the  level  of  knowledge  required  for  effectively 
aiding  users  is  an  important  issue  for  intelligent 
interfaces.  For  example,  in  the  development  of 
expert-consultant  systems  both  Hillman  (1985)  and  Kidd  and 
Cooper  (1985)  have  argued  that  the  success  of  the  system 
depends  on  the  match  between  the  user's  and  system's 
knowledge.  Currently,  many  of  the  research  projects  on 
intelligent  interfaces  focus  solely  on  developing 
sophisticated  knowledge  and  reasoning  structures.  While 
activity  like  this  is  necessary  to  expand  knowledge  about 
intelligent  systems,  there  is  the  danger  that  state  of  the 
art  research  on  intelligent  systems  may  over  design  the 
interface.  The  philosophy  adopted  in  this  research  follows 
the  design  approach  advanced  by  Williges,  Williges,  and 
Elkerton  (in  press),  whereby  explicit  behavioral  objectives 
are  established  for  the  iterative  development  of  a  software 
interface.  This  method  avoids  over  design  and  only  provides 
the  necessary  and  sufficient  knowledge  for  aiding  or 
adapting  to  the  user. 
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In  addition  to  the  issues  of  representing  user  knowledge 
and  skills,  there  are  questions  of  how  to  communicate 
effectively  in  an  intelligent  human-computer  system. 
Indeed,  with  the  introduction  of  intelligence  into  the 
interface  a  true  human- computer  dialogue  exists  and  should 
be  investigated  thoroughly  from  a  human  factors  perspective. 
Of  primary  importance  in  intelligent  dialogue  is  the 
question  of  control  (Rissland,  1984).  More  specifically, 
who  should  initiate  the  dialogue:  the  human,  the  computer, 
or  both. 


Learning  Support  for  Inexperienced  Users 


An  understanding 

of 

the 

learning 

difficulties 

and 

processes  faced  by  users 

of 

software 

interfaces  may 

be 

helpful  when  designing 

an 

online  aid. 

Unfortunately, 

the 

learning  difficulties  and  processes  that  users  encounter  are 
very  complex.  As  Mack,  Lewis,  and  Carroll  (1983)  have 
described,  the  user  is  often  overwhelmed  by  the  difficulty 
of  the  task,  lacks  the  basic  knowledge  of  computers,  often 
makes  ad  ftoc  interpretations,  frequently  generalizes  from 
previous  non-computing  knowledge,  has  difficulty  following 
directions,  does  not  understand  complex  interface  features, 
has  trouble  using  online  help,  and  fails  to  realize  that 
problems  can  interact.  From  this  list  it  is  obvious  that 
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research  addressing  the  training  issues  of  software 
interfaces  is  necessary.  However,  with  all  these  problems 
where  can  online  training  and  user  support  begin?  The  work 
of  Carroll  and  his  colleagues  on  learning  by  doing, 
thinking,  and  knowing  (Carroll  and  Mack,  1984)  provides  a 
convenient  framework  for  addressing  this  question.  Put 
simply,  a  training  interface  should  support  the  user  in 
performing  the  actual  computer-based  task  and  should 
encourage  the  user  to  exploit  past  and  current  knowledge  for 
understanding  the  computer  system. 

Support  for  learning  by  doing.  Carroll  and  his 
colleagues  have  demonstrated  that  inexperienced  users  of 
computer  systems  often  do  not  read  the  documentation  or 
follow  the  training  materials  when  learning  to  use  a 
software  interface  (Carroll  and  Mack,  1984;  Mack,  Lewis,  and 
Carroll,  1983).  Rather,  inexperienced  users  strike  out  on 
their  own,  learning  by  doing  a  computer-based  task. 
Exploiting  active  learning  with  a  software  interface  is  an 
attractive  embedded  training  approach.  From  a  psychological 
perspective,  embedded  training  offers  the  potential  for 
maximum  transfer  of  training  since  the  stimuli  and  responses 
are  highly  similar  in  the  learning  and  transfer  tasks. 
Moreover,  training  individuals  during  the  computer-based 
task  is  appealing  since  meaningful  work  can  be  accomplished 


8 


by  the  user  rather  than  spending  time  in  a  class  or  reading 
a  manual. 

Ideally,  the  training  interface  should  encourage 
exploration  (Mack,  Lewis,  and  Carroll,  1983)  and  hypothesis 
testing  (Shrager  and  Klahr,  1983)  by  the  new  user.  However, 
supporting  learning  by  doing  is  not  easy.  When  exploring 
the  computing  environment,  inexperienced  users  of  computer 
systems  often  commit  errors  that  prevent  meaningful  learning 
at  the  interface.  In  addition,  novice  users  may  get 
hopelessly  lost  due  to  the  complexity  of  the  software 
interface.  To  circumvent  these  problems,  the  interface 
could  be  redesigned  to  block  the  error  states  of  the  novice 
user  (Carroll  and  Carrithers,  1984)  or  the  software  could 
provide  guidance,  error  correction,  and  feedback  to  the 
novice  user  (Carroll  and  Kay,  1985). 

The  development  of  a  "training  wheels"  interface  by 
Carroll  and  Carrithers  (1984)  attempted  to  support  early 
learning  of  a  word  processor  by  blocking  advanced  functions 
and  allowing  only  the  most  basic  word-processing  operations 
(creating  ard  printing  documents).  Thus,  a  new  user  was 
able  to  explore  the  interface  while  doing  meaningful  tasks 
without  entering  into  exasperating  error  states.  Carroll 
and  Carrithers  found  that  more  users  of  the  training  wheels 
interface  completed  a  letter  printing  task  and  did  so  faster 
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than  the  users  of  the  unmodified  word  processor.  Carroll 
and  Carrithers  explained  the  success  of  the  training  wheels 
design  through  users  gaining  feedback  when  attempting  to 
select  blocked  functions  without  suffering  the  negative 
consequences  of  these  potential  error  states.  In  addition, 
Carroll  and  Carrithers  noted  that  the  training  wheels 
interface  may  be  a  more  effective  learning  environment  since 
the  number  of  system  functions  was  limited  and  allowed  the 
users  to  form  and  test  alternative  hypotheses  about  the 
word-processing  system. 

Expanding  on  the  success  of  the  training  wheels  approach, 
Carroll  and  Kay  (1985)  have  implemented  and  evaluated 
several  "scenario  machines"  where  a  new  user  of  a 
word-processing  system  is  guided  through  a  single  path  or 
scenario  for  creating  and  printing  a  document.  The  goal  of 
this  research  was  to  determine  what  combination  of  user 
prompting,  feedback,  and  error  correction  facilitated 
training  and  transfer  performance  in  a  word-processing  task. 

Overall,  the  results  from  Carroll  and  Kay  (1985)  were  in 
agreement  with  the  training  wheels  research  (Carroll  and 
Carrithers,  1984).  Scenario  machines  which  guided  the  user 
down  one  path  for  creating  and  printing  a  document  reduced 
training  time  significantly  when  compared  to  a  control  group 
that  received  an  unmodified  word  processor.  However, 
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transfer  performance  on  a  document  revision  task  was  not 
found  to  differ  between  users  that  received  a  scenario 
machine  and  users  that  received  a  full  word  processor.  In 
fact,  the  results  from  the  scenario  machine  with  prompting 
and  feedback  suggest  that  some  training  implementations  may 
be  damaging  to  transfer  performance.  Specifically,  Carroll 
and  Kay  (1985)  illustrated  that  a  scenario  machine  with 
prompts  and  feedback  for  user  actions  may  destroy  the 
coherence  of  the  computing  task. 

As  a  result  of  these  initial  studies,  questions  about  how 
a  training  interface  should  be  designed  and  how  the  design 
affects  transfer  performance  still  remain  unanswered. 
Clearly,  a  simplified  interface  can  reduce  training  for  new 
users,  but  how  does  this  training  affect  later  performance? 
The  results  from  the  scenario  machine  research  (Carroll  and 
Kay,  1985)  suggest  that  the  training  interface  must  be 
developed  carefully  so  as  to  avoid  changing  the 
computer-based  task  with  the  training  dialogue.  Moreover, 
it  is  interesting  to  speculate  whether  the  simplicity  of  the 
training  wheels  design  will  help  or  hinder  later  performance 
on  a  complex  computer  system.  Will  gradual  exposure  in  a 
stagewise  fashion  facilitate  long-term  performance  or  would 
complete  immersion  with  carefully  constructed  instruction  on 
the  complex  computer  system  be  superior?  Finally,  the  time 
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scale  of  transfer  performance  also  needs  to  be  investigated. 
Both  Carroll  and  Carrithers  (1984)  and  Carroll  and  Kay 
(1984)  studied  early  learning  of  computer  interfaces  that 
did  not  exceed  one  day  in  length.  Therefore,  design  and 
evaluation  of  training  interfaces  should  be  carried  out  over 
a  longer  time  span. 

Support  for  learning  by  knowing  and  thinking.  In 
addition  to  learning  by  doing,  new  users  spend  a 
considerable  part  of  their  time  reading  and  thinking  about  a 
software  interface.  New  users  are  often  faced  with  an 
enormous  amount  of  information  in  the  form  of  manuals, 
online  help,  and/or  tutorials.  Integrating  this  information 
with  any  previous  knowledge  or  skills  is  a  complex  cognitive 
task. 

Much  has  been  written  on  the  theoretical  value  of  models 
(Halasz  and  Moran,  1982),  analogies  (Rumelhart  and  Norman, 
1981)  and  metaphors  (Carroll  and  Mack,  1985)  for  aiding  the 
user  to  learn  and  understand  a  computer  interface.  For 
example,  Rumelhart  and  Norman  (1981)  have  suggested  that 
multiple  analogies  may  be  useful  for  explaining  the 
different  characteristics  of  a  software  interface. 
Subsequently,  Halasz  and  Moran  (1982)  argued  against  this 
approach  stating  that  multiple,  literal  metaphors  may  fail 
to  communicate  a  consistent  structure  to  new  users.  Halasz 
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and  Moran  suggested  the  use  of  new  and  abstract  models  to 
present  an  appropriate  conceptual  base  for  reasoning  about  a 
computer  system.  To  further  complicate  matters,  Carroll  and 
Mack  (1985)  recently  have  argued  against  both  of  these 
approaches  stating  that  metaphors  should  be  used  actively, 
serving  to  supply  users  open-ended  comparisons  suited  to 
their  own  goals.  Specific  analogies  or  conceptual  models, 
Carroll  and  Mack  argue,  are  passive  tools  that  do  not 
encourage  exploration,  and  therefore,  may  not  be  helpful  to 
new  users. 

What  can  be  gleaned  from  these  opposing  theoretical 
viewpoints?  Specifically,  three  methods  for  aiding  novice 
users  with  models,  analogies,  or  metaphors  exist  by: 

1.  exploiting  prior  user  knowledge, 

2.  organizing  new  interface  knowledge,  and 

3 .  encouraging  user  thinking  about  the  software 
interface . 

Initial  research  on  these  methods  appears  to  support  all 
three  approaches.  Consequently,  the  theoretical  efforts, 
although  in  conflict,  have  generated  ideas  which  in  practice 
are  useful  for  assisting  and  instructing  new  computer  users. 

The  first  method  for  supporting  the  user  in  learning  by 
knowing  and  thinking  rests  on  developing  training  and 
documentation  materials  that  exploit  any  prior  user 
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knowledge.  Well  placed  models,  examples,  analogies,  or 
metaphors  may  stimulate  the  new  user  to  understand  the 
software  interface  in  terms  of  other  familiar  objects  or 
tasks.  Probably  the  best  support  for  this  method  can  be 
found  in  the  research  of  Mayer  (1981)  on  presenting  concrete 
models  for  learning  programming  concepts  in  BASIC.  The 
concrete  model  used  for  BASIC  consisted  of  an  input  window, 
a  memory  scoreboard,  a  program  list  and  arrow,  and  an  output 
pad.  Comparison  of  learners  receiving  a  verbal  manual  with 
a  concrete  model  of  BASIC  and  learners  receiving  only  a 
manual  indicated  that  a  concrete  model  could  enhance  a 
learner's  ability  to  generate  and  interpret  moderately 
complex  computer  code.  These  transfer  problems  required 
learners  to  go  beyond  the  instructions  provided  in  the 
manual  to  create  and  interpret  new  programs. 

Similar  results  also  were  presented  by  Mayer  (1981)  for  a 
file-management  language  with  a  concrete  model  consisting  of 
a  file  cabinet,  sorting  baskets,  a  memory  scoreboard,  and  an 
output  pad.  Learners  receiving  the  concrete  model  were 
better  at  creating  file-management  programs  that  went  beyond 
the  simple  programs  presented  in  the  manual.  The 
conclusions  reached  from  these  investigations  by  Mayer  imply 
that  concrete  models  can  be  used  for  enhancing  the  learner's 
ability  to  transfer  skills  creatively  to  new  and  more 
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complex  problems.  Unfortunately,  systematic  methods  for 
identifying  prior  knowledge  (scoreboards  and  file  cabinets) 
still  remain  to  be  developed. 

Beyond  exploiting  previous  user  knowledge,  there  is  also 
a  need  for  developing  procedures  to  enhance  and  convey  the 
organization  of  knowledge  surrounding  the  software 
interface.  Novice  users  cannot  be  expected  to  organize  and 
assimilate  every  aspect  of  the  software  interface.  For 
example,  the  research  of  Mayer  (1981)  has  demonstrated  that 
a  function  of  concrete  models  is  to  organize  the  information 
for  easy  assimilation.  Mayer  provided  concrete  models  of 
BASIC  before  and  after  receiving  the  manual  and  found  an 
increased  recall  jf  technical  and  conceptual  information  for 
tha  before  group.  Mayer  hypothesized  that  the  increased 
recall  was  due  to  an  advanced  organizer  effect.  That  is, 
receiving  the  concrete  model  before  the  manual  allowed 
learners  to  organize  technical  and  conceptual  information  in 
the  manual  for  later  recall.  Thus,  presenting  new  knowledge 
to  users  requires  thoughtful  organization  and  sequencing  of 
training  materials. 

Sullivan  and  Chapanis  (1983)  have  demonstrated  that  the 
new  knowledge  needed  by  a  user  can  be  organized  in  a  manual 
using  well-known  writing  principles  (e.g.,  use  simple, 
familiar  language  and  use  short  active  sentences),  adhering 
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to  systematic  task  analytic  methods  (e.g.,  identify  operator 
tasks  and  determine  sequences  of  operations) ,  and  designing 
iteratively.  The  application  of  these  methods  resulted  in 
an  easy  to  use  text-editing  manual,  increased  the  use  of 
text-editing  commands,  and  improved  the  quality  of  material 
copied  and  edited  during  evaluation. 

Similar  results  have  been  found  by  Foss,  Rosson,  and 
Smith  (1982)  with  a  text-editing  manual  that  attempted  to 
take  into  consideration  the  limited  information-processing 
capabilities  of  a  novice  user.  Foss,  et  aX.  (1982) 
hypothesized  that  the  original  text-editing  manual 
overwhelmed  users  with  too  much  abstract  detail  for 
effective  use.  The  manual  for  the  text  editor  was  rewritten 
to  present  only  enough  information  for  simple  command 
execution.  The  increased  speed  and  decreased  errors  during 
document  editing  and  creation  indicated  that  manuals  can  be 
redesigned  effectively  from  a  cognitive  perspective 
emphasizing  reduced  memory  load. 

Formal  development  of  principles  for  effective  manual 
design  have  also  been  proposed  by  Carroll  (1984)  wit.i  the 
"minimalist"  philosophy.  In  proposing  and  testing  minimal 
manuals,  Carroll  has  emphasized  that  manuals  should  have 
less  text,  have  a  greater  task  orientation,  provide  for  more 
learner  initiative,  provide  more  error  recovery  information. 
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and  serve  as  a  reference  after  training.  Preliminary 
testing  showed  a  40%  reduction  in  training  time  with  the 
minimal  manual. 

A  final  graphical  method  for  organizing  knowledge  about 
the  user  interface  has  been  described  by  Sebrechts,  Deck, 
and  Black  (1983).  These  researchers  have  developed  a 
diagrammatic  approach  for  describing  the  knowledge  and 
procedures  required  for  using  commands  in  a  software 
interface.  The  diagrams  proposed  were  motivated  by 
psychological  research  on  cognitive  maps  and  resembled 
flowcharts  for  the  selection  of  commands.  Initial 
evaluation  of  the  flowcharts  as  an  instructional  device  when 
compared  with  a  textual  manual  indicated  an  increased  speed 
and  accuracy  for  users  of  the  diagrams.  Unfortunately,  as 
with  the  minimal  manual  (Carroll,  1984),  statistical 
evidence  for  this  evaluation  was  not  available. 

From  these  initial  investigations,  there  appear  to  be 
several  methods  available  for  organizing  the  new  knowledge 
required  for  using  a  software  interface.  These  techniques 
ranged  from  a  systems  design  method  (Sullivan  and  Chapanis, 
1982),  to  a  cognitive  design  approach  (Foss,  Rosson,  and 
Smith,  1982),  to  an  instructional  technology  method 
(Sebrechts,  Decker,  and  Black,  1983).  Interestingly,  all 
methods  emphasized  simplicity  when  describing  the 
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organization  of  the  software  interface.  Indeed,  a 
motivating  factor  for  using  models,  analogies,  or  metaphors 
for  aiding  the  naive  user  is  the  simplification  of  a 
potentially  complex  interface. 

However,  a  simplified  model  of  a  software  interface  could 
lead  to  novices  failing  to  learn  advanced  features  of  the 
system.  The  very  low  probability  that  any  model  can 
describe  all  interface  features  adds  to  these  learning 
difficulties.  Consequently,  a  third  method  for  helping  new 
users  requires  the  model,  analogy,  or  metaphor  to  encourage 
active  mental  exploration  of  the  software  interface  (Carroll 
and  Mack,  1985).  The  benefits  of  encouraging  mental 
exploration  can  be  seen  in  the  research  of  Mayer  (1981). 

Mayer  (1981),  in  addition  to  studying  the  effects  of 
concrete  models  on  learning  computer  programming,  also 
investigated  the  effects  of  self-elaboration  by  the  learner. 
Mayer  provided  for  elaboration  in  several  fashions.  First, 
learners  were  asked  to  relate  information  from  the  manual  to 
concrete  models.  Next,  learners  were  asked  to  compare  and 
contrast  programming  statements.  Finally,  learners  were 
asked  to  take  notes  on  a  programming  lecture.  The  results 
indicated  that  elaboration  increased  a  learner’s  ability  to 
transfer  programming  skills  creatively  to  problems  beyond 
what  is  covered  in  a  manual  or  lecture.  Moreover,  the 
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elaboration  activity  increased  a  learner's  retention  of 
technical  and  conceptual  information. 

To  summarize  the  elaboration  research  of  Mayer  (1981), 
software  users  should  be  encouraged  to  put  information  in 
their  own  words.  This  may  be  done  explicitly  through 
instructions  or  through  concrete  models  supplied  to  the 
user.  Despite  these  suggestions,  however,  selection  of 
elaboration  procedures  is  still  difficult.  Little  is  known 
about  what  type  of  models,  analogies,  or  metaphors  will 
encourage  elaborative  processing  or  whether  explicit 
directions  to  elaborate  will  be  followed  in  an  actual 
software  interface. 

Models  for  Aiding  Information  Retrieval 

Analysis  of  the  previous  topics  on  intelligent 
human-computer  interfaces  and  supporting  the  inexperienced 
user  suggests  that  methods  and  procedures  for  representing 
information  about  the  user,  task,  or  computer  is  of  prime 
importance.  To  aid  the  inexperienced  user,  skills  and 
capabilities  must  be  referenced  to  computer-based 
procedures.  In  this  research  an  approximation  method  was 
adopted  to  determine  how  much  and  what  type  of 
command-selection  data  must  be  represented  to  aid 
inexperienced  users. 
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Specifically,  three  levels  of  command- selection  advice 
were  constructed  for  inexperienced  users.  The  first  online 
aid  was  based  on  a  frequency  profile  of  the  commands 
experienced  users  selected.  The  second  online  aid  was  a 
sequence  model  of  experienced  users'  transitions  from  one 
search  procedure  to  another.  Finally,  the  th_rd  online  aid 
was  a  planning  model  of  experienced  users*  command  selection 
that  attempted  to  take  into  consideration  the  more  cognitive 
aspects  of  the  computer-based  task.  Progressing  from  the 
frequency  to  sequence  to  planning  models  of  command 
selection  provides  additional  and  potentially  valuable 
information  to  the  inexperienced  user.  However,  in  certain 
interface  design  situations,  the  cost  and  difficulty  of 
constructing  the  more  advanced  command-selection  models  may 
be  prohibitive.  Thus,  the  intent  of  the  research  was  to 
determine  what  level  of  command-selection  information  was 
useful  for  aiding  the  novice  user. 

Frequency  ynodels .  One  of  the  most  basic  models  of  a  user 
and  a  task  is  ?  frequency  ^profile  of  command  usage. 
Monitoring  the  command  selection  of  users  has  been 
undertaken  previously  (Boies,  1^74;  Good,  1985;  Hanson, 
Kraut,  and  Farber,  1984;  Whiteside,  Archer,  Wixon,  and  Good, 
1982).  However,  these  efforts  have  been  devoted  towards 
improving  the  design  of  conventional  software  interfaces 
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(i.e.,  command  languages  and  text  editors).  In  the  context 
of  this  dissertation,  command  frequency  profiles  were  used 
to  provide  command-selection  advice  to  inexperienced  users. 

A  frequency  model  has  been  implemented  and  evaluated  by 
Elkerton  and  Williges  (1984b,  1985).  These  investigators 
found  that  a  profile  methodology  containing  frequency-based 
information  on  the  selection  of  search  procedures  by 
experienced  users  could  be  used  to  train  novice  users.  The 
frequency  model  was  an  easily  constructed  performance  model 
that  potentially  could  be  used  in  highly  repetitive  tasks. 

There  were  some  limitations  of  this  modeling  approach 
reported  by  Elkerton  and  Williges  (1984b,  1985).  First,  the 
frequency-based  model  could  not  provide  command- sequence 
information.  The  selection  of  advice  was  not  sensitive  to 
use  of  other  commands.  Second,  the  model  could  not  supply 
inexperienced  users  explanations  why  a  search  procedure  was 
suggested  or  how  to  use  a  particular  search  procedure. 
Finally,  the  idiosyncrasies  of  novice  file  search  could  not 
be  accounted  for  by  the  model.  Inexperienced  users  that 
prefer  a  search  procedure  and  perform  adequately  with  it 
should  be  allowed  to  use  it.  The  model  should  be  capable  of 
learning  or  adapting  to  an  individual. 

Due  to  these  potential  limitations,  Elkerton  and  Williges 
(1984b)  describe  the  file  search  assistant's  communication 
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with  the  inexperienced  user  as  intrusive  since  file  search 
efficiency  degraded  on  trials  in  which  advice  was  given. 
However,  despite  these  problems  the  frequency-based  model 
was  successful  in  changing  an  inexperienced  user's  search 
strategy.  That  is,  the  frequency-based  model  was  not 
successful  as  an  assistant,  but  demonstrated  some  success  as 
an  instructional  device. 

Why  might  frequency-based  models  of  command  selection  be 
useful  for  inexperienced  users?  First,  novice  users  may  not 
need  sophisticated  methods  and  advice  when  using  a  software 
interface.  Some  novices  may  only  require  hints  or 
suggestions  to  break  out  of  inefficient  search  strategies. 
Second,  frequency-based  models  are,  by  definition, 
incomplete  models  of  command  selection,  and  therefore, 
encourage  the  active  learning  (Carroll  and  Mack,  1985)  and 
elaborative  processing  (Mayer,  1981)  that  are  useful  when 
learning  a  software  interface. 

Sequence  models.  As  noted  by  Elkerton  and  Williges 
(1984b)  sequence  information  is  one  potential  improvement 
over  a  frequency-based  model.  These  models  attempt  to 
capture  the  context  of  command  selection  by  considering  past 
and  present  commands  in  predicting  future  commands.  A  good 
example  of  the  potential  for  including  sequence  information 
exists  in  hierarchical  file-management  tasks.  Spine, 
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Whiteside,  and  Williges  (1984)  and  Hanson,  Kraut,  and  Farber 
(1984)  demonstrated  that  users  frequently  follow  a  command 
to  move  to  a  new  directory  by  a  command  to  list  the  files  in 
the  new  directory.  Sequence  information  such  as  this  would 
be  valuable  to  inexperienced  users  who  are  attempting  to 
navigate  and  find  information  in  a  hierarchical  file  system. 

As  with  analysis  of  frequency  data,  sequence  analysis  has 
aimed  primarily  at  the  design  of  conventional  human-computer 
interfaces.  These  research  endeavors  include  the  analysis 
of  keystroke  and  state  transitions  in  text  editing 
(Whiteside,  Archer,  Wixon,  and  Good,  1982),  command 
transitions  in  an  operating  system  (Hanson,  Kraut,  and 
Farber,  1984)  and  the  Markov  processes  of  selecting  database 
retrieval  commands  (Penniman,  1975).  These  investigations 
have  shown  a  variety  of  methods  for  representing  and 
analyzing  sequence  information.  The  most  basic  form  of 
sequence  information  is  contained  in  a  transition  matrix 
which  lists  the  probability  of  moving  from  one  command  to 
another  at  some  future  point  in  the  dialogue.  As 
illustrated  by  Whiteside,  Archer,  Wixon,  and  Good  (1982) 
various  orders  of  transition  matrices  can  be  constructed  to 
predict  beyond  the  next  keystroke  or  editing  state.  In 
addition,  Hanson,  Kraut,  and  Farber  (1984)  have  shown  that 
transition  matrices  may  also  be  subjected  to  multivariate 
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statistical  procedures  like  hierarchical  cluster  analysis 
and  principal  components  analysis  to  extract  important 
sequence  structures  in  command  usage. 

Before  implementing  empirical  sequence  models  for  aiding 
novice  users,  research  must  be  undertaken  focusing  on  the 
benefits  over  frequency-based  representations.  The 
additional  cost  and  time  of  a  sequential  analysis  may  not  be 
justifiable  in  some  applications.  In  addition,  from  a 
behavioral  perspective  sequence  models  may  be  difficult  to 
comprehend  by  the  user.  Adding  sequence  information 
increases  the  complexity  of  the  model  by  the  square  of  the 
number  of  commands  in  a  frequency  model. 

Planning  models .  The  third  model  for  assisting  and 
instructing  users  goes  beyond  empirical  sequence 
representations  by  making  assumptions  about  the  cognitive 
processing  of  the  user.  The  construction  of  plan-based 
models  assumes  that  inexperienced  users  must  problem  solve 
with  computer-based  procedures.  To  use  Norman's  (1984)  four 
stages  of  user  activity,  the  inexperienced  user  must  form 
mental  intentions  and  actions,  execute  these  plans  with 
automated  procedures,  and  finally  evaluate  the  result. 

Norman  (1984)  suggests  a  variety  of  conventional  methods 
like  memory  aids,  menus,  and  workbenches,  to  support  user 
stages.  However,  in  this  dissertation  the  development  of 
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plan-based  models  for  assisting  and  instructing 
inexperienced  users  focused  on  the  selection  and  execution 
of  actions.  The  importance  of  aiding  action  selection  and 
execution  is  underscored  by  the  error  rate  during  these 
stages.  In  the  UNIX  screen-based  text  editor,  Riley  and 
O'Malley  (1984)  have  estimated  from  empirical  data  that  58% 
of  user  errors  could  be  categorized  as  selection  and 
execution  errors.  Therefore,  as  Jackson  and  Lefrere  (1984) 
suggest,  a  plan-based  system  should  be  capable  of 
criticizing  and  providing  developmental  support  for  a  user's 
selection  and  execution  of  plans. 

What  is  a  plan?  Briefly,  a  plan  can  be  conceived  of  as  a 
series  of  mental  steps  towards  the  solution  of  a  problem 
(Schank  and  Abelson,  1977).  The  steps  are  defined  as 
cognitive  activity  in  an  attempt  to  describe  the  underlying 
processes  of  behavior.  Homogeneous  chunks  of  mental 
behavior  can  be  captured  in  plans  for  specific  circumstances 
and  individuals.  Card,  Moran,  and  Newell  (1983)  and 
Robertson  and  Black  (1983)  have  both  demonstrated  that  plans 
are  cognitive  entities  chunked  together  in  temporal  and 
functional  sequence.  This  chunking  of  mental  behavior  often 
results  from  the  individual  facing  large,  complex  problems 
that  are  too  difficult  to  be  solved  with  one  mental 
operation.  To  overcome  these  short-term  memory  constraints. 
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the  learner  often  employs  a  "divide  and  conquer”  strategy, 
planning  problem  solving  as  small  sets  of  subtasks  (Card, 
Moran,  and  Newell,  1983). 

As  a  consequence  of  the  cognitive  nature  of  plans,  these 
models  of  behavior  will  always  be  approximate.  Plans  cannot 
be  observed  directly.  However,  different  planning  processes 
do  exist  and  need  to  be  understood  before  models  can  be 
built  for  aiding  novice  users.  The  three  types  of  planning 
processes  that  may  influence  the  development  and 
implementation  of  planning  models  include: 

1.  compiled  planning, 

2.  well-specified  planning,  and 

3.  opportunistic  planning. 

Compiled  planning  is  linear  in  nature.  An  individual 
selects  a  well-practiced  method  for  achieving  a  goal  and 
executes  the  method  without  interruption.  Compiled  planning 
resembles  the  automatic  processing  elaborated  on  by 
Schneider  and  Shiffrin  (1977)  and  the  script-based 

processing  described  by  Schank  and  Abelson  (1977).  A 
compiled  plan  requires  little  attention  for  execution  and 
exploits  task-specific  knowledge  learned  through  extensive 
practice. 

Examples  of  compiled  plans  abound  in  the  use  of 

computer-based  systems.  One  plan  that  most  users  compile  is 
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the  command  sequence  for  logging  on  to  a  computer  system. 
At  first,  the  logging  sequence  may  be  a  difficult 
problem-solving  task  (Carroll  and  Mack,  1984).  However, 
with  practice  the  user  internalizes  the  sequence  and 
executes  it  without  thinking.  Recent  research  suggests  that 
experienced  users  may  develop  and  exploit  compiled  plans  in 
domains  such  as  programming  (Ehrlich  and  Soloway,  1984)  and 
text  editing  (Card,  Moran,  and  Newell,  1983;  Robertson  and 
Black,  1983). 

Well-specified  planning  can  be  distinguished  from 
compiled  plans  in  that  the  sequence  of  mental  operators  must 
be  decided  upon  explicitly  rather  than  existing  as  a 
cherent  sequence  of  mental  operations.  Consequently,  the 
controlled  processing  (Schneider  and  Shiffrin,  1977)  of 
well-specified  plans  requires  substantial  mental  workload 
during  plan  formation.  Circumstances  requiring 
well-specified  planning  include  critical  instances  where 
mistakes  can  be  disastrous  and  complex  or  novel  problems 
where  planning  will  speed  solution. 

While  the  two  previous  planning  models  focused  almost 
entirely  upon  plan  formation  and  execution,  opportunistic 
planning  focuses  to  a  greater  extent  on  plan  evaluation  and 
revision  (Hayes-Roth  and  Hayes-Roth,  1979;  Jackson  and 
Lefrere,  1984).  With  opportunistic  planning,  humans  are 


27 


assumed  to  be  using  the  information  acquired  during  plan 
execution  to  monitor  and  revise  plans.  Thus,  opportunistic 
planning  is  much  more  dynamic  than  compiled  or 
well-specified  planning.  Data  supporting  opportunistic 
planning  in  computing  systems  can  be  found  m  text-editing 
behavior.  By  analyzing  command  transitions,  Whiteside, 
Archer,  Wixon,  and  Good  (1982)  have  revealed  that  users 
frequently  alternate  between  text-editing  subtasks.  After 
leaving  an  editing  state,  users  tend  to  return  to  it  after  a 
single  intervening  state. 

Research  efforts  devoted  to  developing  planning  aids  for 
users  of  computer  systems  have  attempted  to  support  all 
three  planning  processes  (compiled,  well-specified,  and 
opportunistic).  The  help  system  described  by  Finin  (1982) 
attempted  to  diagnose  the  sometimes  opportunistic  and 
potentially  erroneous  command  sequences  of  inexperienced 
users  of  a  computer  operating  system.  For  example,  if  a 
user  attempted  to  erase  all  previous  versions  of  a  file  by 
using  multiple  DELETE  commands,  then  the  planning  aid  would 
suggest  use  of  the  PURGE  command.  Similar  help  and  advisory 
systems  have  been  implemented  for  computer-aided  design 
(Cullingford,  Krueger,  Selfridge,  Bienkowski,  1982),  file 
management  in  UNIX  (Wilensky,  Arens,  and  Chin,  1984),  and 
automated  symbolic  integration  with  MACSYMA  (Genesereth, 
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1982).  The  goals  of  these  planning  systems  have  not  been  to 
identify  and  research  specific  planning  processes,  but  to 
provide  context-sensitive  advice  without  overburdening  the 
short-term  memory  constraints  of  the  user  (Darlington, 
Dzida,  and  Herda,  1983). 

To  identify  ana  support  compiled,  well-specified,  and 
opportunistic  planning  will  require  the  construction  of 
systematic  methods  for  extracting  plans  from  computer  users. 
Briefly,  the  method  for  extracting  information  will  depend 
on  the  planning  process.  For  instance,  extracting  compiled 
plans  from  users  will  probably  depend  on  performance 
methodologies  like  path  analysis  (Alty,  1984,  Tolle  and  Hah, 
1985),  or  chronometric  analysis  (Robertson  and  Black,  1983). 
The  reason  for  relying  on  performance  rather  than  more 
subjective  verbal  reports  rests  in  the  information 
processing  of  compiled  plans.  As  stated  earlier,  compixed 
plans  are  executed  in  an  automatic  fashion  (Schneider  and 
Shiffrin,  1977)  and  may  not  enter  short-term  memory. 
Computer  users  may  not  be  able  to  report  compiled  plans 
accurately  (Ericcson  and  Simon,  1984). 

In  contrast,  well-specified  plans  are  under  the  direct 
controlled  processing  of  the  user  and  will  result  in  the 
information  being  attended  to  in  short-term  memory.  Verbal 
report  procedures  (Ericcson  and  Simon,  1984),  structured 
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sorting  procedures  (Tullis,  1985),  or  similarity  rating 
procedures  (Kay  and  Black,  1984)  may  be  used  to  extract  this 
information  from  a  user's  memory. 

Finally,  opportunistic  planning  poses  difficulties  for 
both  performance  and  subjective  assessment  since  users  may 
not  have  established  plans  for  many  computing  tasks.  The 
designer  of  planning  models  probably  will  have  to  adopt  a 
formal,  analytical  method  for  constructing  these  plans  using 
subject  matter  experts  (e.g..  Card,  Moran,  and  Newell, 
1984) . 

Pi alocrues  for  Aiding  Information  Retrieval 

Once  knowledge  has  been  captured  and  represented  in  a 
software  interface,  it  must  be  presented  to  the  user. 
However,  before  presenting  command-selection  information  to 
inexperienced  users,  decisions  must  be  made  on  who  should 
control  the  dialogue.  In  this  dissertation,  the  three 
dialogue  initiatives  to  be  evaluated  for  an  online  aid  are 
user-initiated,  computer-initiated,  and  mixed-ini tiated 
(both  user-  and  computer-initiated)  dialogues.  Only  limited 
human  factors  data  exists  for  implementing  these  dialogues. 
The  conventional  wisdom  has  been  to  provide  the  user  with 
control  of  the  dialogue  (Williges  and  Williges,  1984). 
However,  the  assumption  of  user  control  may  be  open  to 
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question  with  the  increased  intelligence  in  software 
interfaces  and  the  increased  emphasis  of  online  training  for 
inexperienced  users. 

User-initiated  dj  alocrues .  With  an  online  aid  or  help, 
some  evidence  does  exist  for  providing  standard 
user-initiated  dialogues.  For  example.  Cohill  and  Williges 
(1985)  evaluated  eight  online  help  conditions  by 
manipulating  the  initiation  of  help  (user  versus  computer), 
the  selection  of  help  topic  (user  versus  computer),  and  the 
format  of  help  (online  versus  hard  copy).  From  this 
experiment.  Cohill  and  Williges  found  that  user  initiated 
and  selected  help  facilitated  browsing  and  exploration  by 
the  user.  Furthermore,  user  initiated  and  selected  help 
decreased  the  total  time  on  the  editing  task,  errors,  and 
command  use  when  compared  to  other  help  groups  and  a  control 
group  that  did  not  receive  help  information.  From  this 
investigation,  the  need  for  user-initiated  dialogues  is 
clear.  Computer  users  are  well  aware  of  their  difficulties 
and,  if  provided  a  well-designed  help  system,  can  search  and 
discover  the  information  required  to  solve  a  computing 
problem. 

Computer- ini tiated  dialogues .  However,  there  are  times 
when  users  may  not  know  of  their  computing  shortcomings. 
Indeed,  the  research  of  Elkerton  and  Williges  (1984b)  on 
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file  search  assistance  revealed  the  search  strategies  used 
by  inexperienced  users  can  be  improved.  Unfortunately,  the 
cost  of  improving  file  search  strategies  was  large  in  terms 
of  the  intrusiveness  of  the  online  assistant.  The  online 
command-selection  assistant  implemented  by  Elkerton  and 
Williges  (1984b)  increased  task  times  when  advice  was  given. 
Clearly,  computer-initiated  dialogues  can  be  disruptive  to 
the  current  computer-based  task. 

Nevertheless,  as  Burton  and  Brown  (1982)  have  shown,  the 
benefits  of  computer-initiated  dialogues  can  be  great  if  the 
conditions  for  interrupting  the  user  are  carefully  selected. 
Burton  and  Brown  (1982)  in  providing  advice  to  students  for 
WEST  (a  game  to  teach  basic  arithmetic  skills)  implemented 
an  advisor  that  was  forgiving  and  only  provided  advice  when 
absolutely  necessary.  More  specifically,  the  computerized 
advisor  only  gave  advice  when  the  students  were  weak  in  a 
specific  arithmetic  skill  and  if  an  appropriate  example  was 
available.  The  advisor  did  not  give  advice  consecutively  or 
when  the  user  committed  careless  errors.  Descriptive 
statistics  indicated  an  increased  use  of  optimal  arithmetic 
strategies  and  skills  with  the  computerized  advisor. 
However,  inferential  statistics  were  not  provided  by  Burton 
and  Brown  (1982).  Further  research  is  required  to  determine 
whether  computer-initiated  advice  can  be  beneficial  to 
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improve  the  skills  of  users  who  may  be  unaware  of  superior 
computer-based  methods. 

iiixcd-ini  +  i  ^ted  di alocr.ies .  Together  ,  the  capability  of 
the  computer  offering  advice  and  the  user  having  the 
opportunity  to  ask  for  advice  is  an  appealing  mixed 
dialogue.  In  skill  acquisition,  mixed-initiative  dialogues 
could  be  valuable  for  suggesting  more  powerful  automated 
procedures  and  providing  for  times  when  the  user  is  at  a 
loss  for  a  method  and  must  request  help.  Likewise, 
mixed-initiative  dialogues  in  information  retrieval  may  be 
valuable  for  calling  attention  to  data  that  could  be  ignored 
by  the  user  (Ehrenreich,  1981).  Despite  these  potential 
advantages  and  several  implementations  of  mixed-initiative 
dialogues  (Cullingford,  Krueger,  Selfridge,  Bienkowski, 
1982;  Finin,  1982;  Grignetti,  Hausmann,  and  Gould,  1975; 
Wilenski,  Arens,  and  Chin,  1984),  the  behavioral  efficacy  of 
these  dialogues  is  unproven. 


PURPOSE 


The  background  literature  on  intelligent  interfaces  and 
previous  research  by  Elkerton  and  Williges  (1984b,  1985) 
suggested  that  models  and  dialogues  are  important  factors 
when  designing  an  online  aid  for  inexperienced  computer 
users.  Therefore,  the  goals  of  this  investigation  were  to 
develop  and  evaluate  frequency,  sequence,  and  planning 
models  of  information  retrieval  to  be  used  as  active 
learning  aids  with  computer-,  user-,  and  mixed-ini tiated 
dialogues . 

Development  and  evaluation  of  these  active  learning  aids 
required  two  experiments.  The  first  experiment  was  devoted 
to  model  development  and  focused  on  the  construction  of  the 
frequency,  sequence,  and  planning  models  of  information 
retrieval.  This  model  development  experiment  required  the 
identification  of  performance  differences  in  information 
retrieval  for  novice  and  experienced  users,  as  well  as  the 
systematic  construction  of  command-selection  models  based  on 
frequency,  sequence,  and  planning  data. 

The  second  experiment  implemented  and  evaluated  the 
command- selection  aids  for  inexperienced  users.  This 
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experiment  attempted  to  describe  any  changes  in  behavior 
that  occurred  as  a  result  of  receiving  the  online  aids  with 
the  computer-,  user-,  or  mixed-ini tiated  dialogues. 
Information  retrieval  performance,  strategy,  and  subjective 
evaluations  were  collected  from  the  aided  users  to  determine 
which  model  and  dialogue  were  least  intrusive  and  possibly 
beneficial  when  provided  to  novices.  Moreover,  performance 
and  strategy  were  also  monitored  in  a  final  unaided  session 
to  determine  any  possible  transfer  of  training  for  the  three 


models. 


EXPERIMENT  Is  DEVELOPMENT  OF  THE  COMMAND- 
SELECTION  MODELS 


Approach 

In  addition  to  the  expert-novice  performance  differences 
which  had  to  be  demonstrated  in  this  experiment,  both  the 
knowledge  and  skills  of  the  information  retrieval  task  had 
to  be  extracted  from  the  more  experienced  subjects. 
Extracting  information  retrieval  knowledge  focused  on 
identifying  the  important  and  meaningful  aspects  of  the  task 
from  a  user's  perspective.  The  knowledge  extraction 

techniques  sought  to  identify  common  sets  of  information 
retrieval  problems  through  the  use  of  psychological  sorting 
and  statistical  clustering  procedures.  The  sorting  and 
clustering  procedures  were  used  to  suggest  a  task 
organization  that  would  be  the  basis  for  development  of  the 
three  command-selection  models.  These  sorting  and 
clustering  procedures  served  to  generalize  the 
command-selection  models  so  that  specific  advice  was  not 
required  for  every  information  retrieval  problem. 

The  model  development  activity  in  this  experiment 

concerned  the  construction  of  the  frequency,  sequence,  and 
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plan-based  command- selection  models.  Conceptually,  these 
three  models  attempted  to  capture  the  information  retrieval 
skills  of  experienced  users  at  three  distinct  levels.  The 
frequency  and  sequence  models  are  performance-based 
representations  that  are  incomplete,  but  simple  to  construct 
models  of  command  selection.  The  plan-based  model  is  more 
complete  in  describing  both  a  sequence  of  search  procedures 
and  a  verbal  description  of  the  search  processes  to  be  used 
during  command  selection.  The  planning  model  attempts  to 
capture  the  more  cognitive  aspects  of  the  information 
retrieval  task  and  as  a  result  is  more  difficult  and  time 
consuming  to  construct.  Both  the  methodological  details  for 
constructing  the  models  and  the  differences  between  novice 
and  experienced  users  for  the  specific  information  retrieval 
task  are  described  in  this  experiment. 

Method 

Subjects.  Twenty- six  university  students  volunteered  and 
were  paid  $50.00  for  participating  in  the  experiment.  Of 
these  subjects,  12  were  classified  as  experienced  and  14  as 
novice  computer  users.  Two  of  the  novice  subjects  were 
excluded  from  the  experiment  for  failure  to  meet  training 
criteria  yielding  a  novice  sample  size  of  12. 
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Subject  classification  was  based  on  previous  background 
experience  with  computers.  Experienced  subjects  (hereafter 
referred  to  as  experts)  were  required  to  have  at  least  two 
years  of  experience  using  interactive  computer  systems, 
while  novice  subjects  were  required  to  have  less  than  20 
total  hours  of  experience  on  any  interactive  computer 
system.  The  background  information  presented  in  Table  1 
illustrates  the  advanced  computer  training  of  the  expert 
subjects  and  the  relative  lack  of  computing  experience  for 
novice  subjects.  In  addition  to  these  experience  criteria, 
all  subjects  were  required  to  complete  and  score  70%  on  a 
recognition  test  of  the  information  retrieval  database  and 
to  be  capable  of  finding  information  in  less  than  300 
seconds  (average)  at  the  end  of  training.  Of  the  two 
novices  excluded  from  the  experiment  for  failure  to  meet 
these  criteria,  one  novice  could  not  find  information  fast 
enough  and  one  novice  was  not  familiar  with  the  information 
retrieval  database. 

All  subjects  were  screened  for  20/30  visual  acuity  and 
familiarity  with  a  Qwerty  keyboard.  The  vision  test  was 
administered  with  a  Bausch  and  Lomb  Vision  Tester.  All 
subjects  indicated  that  they  understood  their  rights  as 
research  participants  by  signing  an  informed  consent  form 
(see  Appendix  A)  before  the  start  of  the  experiment. 
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TABLE  1 

Background  Experience  of  Subjects  in  Experiment  1 


Background  Variable  Median  Range 


Number  of  courses  using 

interactive  systems  8.5  2-15 


Hours/day  using  the 

computer  3  1-7 

Number  of  interactive 

editors  used  4  2-7 

Number  of  other  software 

products  used  (databases,  2.5  0-8 

spread  sheets,  etc.) 

Non-academic  experience  with 

interactive  systems  11  of  12  experts 


Number  of  courses  using 

interactive  systems  1  0-2 


Total  hours  using  the 

computer  10  6-20 

Number  of  interactive 

editors  used  0  0-1 

Number  of  other  software 

products  used  (databases,  0  0-2 

spread  sheets,  etc.) 


Non-academic  experience  with 
interactive  systems 


3  of  12  novices 
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File  hierarchy.  The  information  retrieval  task  in  this 
experiment  required  subjects  to  retrieve  information  from  15 
files  arranged  in  a  three-level  hierarchy.  A  diagram  of 
this  file  hierarchy  is  shown  in  Figure  1.  The  15  files  in 
the  hierarchy  contained  both  textual  and  numerical  data  on 
army  operations,  tanks,  armored  personnel  carriers  (APCs), 
and  combat  support.  Tables,  hierarchies,  lists,  and 
paragraphs  were  used  to  structure  the  information  in  each 
file.  In  total,  the  database  contained  2780  lines  with 
individual  files  having  from  55  to  447  lines.  Portions  of 
the  15  files  in  the  database  are  presented  in  Appendix  B. 

As  mentioned  earlier,  all  subjects  were  given  a 
recognition  test  on  the  file  hierarchy  and  file  content  to 
assure  a  basic  understanding  of  the  database.  The 
recognition  test  was  a  two-part  test.  The  first  part  of  the 
test  required  subjects  to  place  file  labels  on  an  empty 
diagram  like  that  shown  in  Figure  1.  In  this  test,  subjects 
were  given  27  file  labels  (15  correct  labels  and  12  foils) 
and  instructed  to  place  the  file  labels  in  the  correct 
position  on  the  empty  file  diagram.  The  second  part  of  the 
recognition  test  required  subjects  to  complete  a  14  question 
multiple-choice  test  on  database  content  (Appendix  C) . 
Subjects  were  not  allowed  to  view  the  file  hierarchy  during 
this  test.  A  combined  test  score  of  70%  was  required  of 


each  subject. 
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Figure  1.  File  hierarchy  for  the  file  selection  and  search 


task 
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Information  retrieval  hardware  and  software .  The 
information  retrieval  system  which  displayed  data  in  the 
file  hierarchy  was  written  in  FORTRAN  and  implemented  on  a 
VAX  11/750  computer  running  the  VMS  operating  system.  An 
elevated  time-sharing  priority  was  used  during  the 
experiment  to  maintain  the  speed  and  accuracy  of  the 
retrieval  environment.  Two  Digital  VTlOOs  were  used  for  the 
information  retrieval  experiment.  The  primary  VT100  was 
used  for  displaying  the  database,  while  the  secondary  VT100 
was  used  for  presenting  the  search  problems.  The  primary 
display,  as  shown  in  Figure  2,  contained  a  7-line  window 
based  on  the  results  of  Elkerton  and  Williges  (1984a),  a 
file  status  line,  an  input  and  message  line,  a  work  area, 
and  a  touch  keypad.  Subjects  selected  search  procedures 
through  the  touch  keypad  that  was  implemented  with  a  Carroll 
Technology  touch-entry  device.  These  12  search  procedures 
manipulated  the  database  through  the  7-line  window. 
Subjects  used  a  VT100  (Qwerty)  keyboard  for  data  entry  with 
some  of  the  search  procedures.  Finally,  subjects  were 
monitored  on  their  retrieval  behavior  through  embedded 
performance  measurement  (Cohill  and  Ehrich,  1983). 

Search  procedures .  The  12  search  procedures  that 
subjects  were  provided  on  the  touch  keypad  contained  file 
selection  procedures  (ZOOM  IN,  ZOOM  OUT,  and  FILE  SELECT), 
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ballistic  search  procedures  for  large  movements  within  a 
file  (SECTION,  SEARCH,  SEARCH-AND,  SEARCH-AND-NOT,  and 
INDEX),  and  fine  adjustment  search  procedures  for  small 
movements  within  a  file  (SCROLL  UP  and  DOWN,  PAGE  UP  and 
DuWN) . 

The  file  selection  procedures,  ZOOM  IN,  ZOOM  OUT,  and 
FILE  SELECT,  were  used  exclusively  for  traversing  the  file 
hi~  archy.  The  ZOOM  IN  and  ZOOM  OUT  procedures  updated  the 
current  file  in  the  7-line  display  window  by  moving  up  and 
down  the  file  hierarchy  (Figure  1).  Subjects  could  use  the 
ZOOr'i  IN  procedure  to  move  down  the  file  hierarchy  whenever  a 
highlighted  line  appeared  in  the  7-line  window.  Highlighted 
lines  were  the  file  topics  in  the  hierarchy  and  were 
embedded  in  each  of  the  parent  files  of  the  hierarchy. 
Thus,  a  highlighter  line  meant  that  more  information  could 
be  retrieved  on  a  specific  topic  like  TANK  DIVISION  COMBAT 
SUPPORT.  In  contrast,  the  ZOOM  OUT  procedure  could  be  used 
to  move  up  the  file  hierarchy.  Lastly,  tr.e  FILE  SELECT 
procedure  was  used  for  selecting  the  files  more  directly. 
FILE  SELECT  allowed  subjects  to  select  all  files  below  the 
current  file  in  the  hierarchy  through  menu  selection. 

The  ballistic  search  procedures  consisted  of  search 
procedures  that  supplemented  the  organization  of  the  current 
file  (SECTION  and  INDEX)  and  string  search  procedures 
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(SEARCH,  SEARCH-AND,  and  SEARCH-AND-NOT)  for  electronically 
scanning  the  current  file.  The  SECTION  procedure  exploited 
and  presented  the  hierarchical  structure  of  each  file  in  the 
7-line  window.  The  SECTION  procedure  presented  a  menu  of 
sections  and  subsections  to  the  subject  in  the  work  area  of 
the  primary  display.  Choosing  one  of  these  sections  would 
position  the  current  file  at  the  appropriate  location  in  the 
7-line  window.  Conceptually,  the  SECTION  procedure 
resembled  a  table  of  contents. 

The  INDEX  procedure  also  used  the  work  area  of  the 
primary  display  when  presenting  an  alphabetical  list  of 
topics  and  subtopics  for  the  current  file.  The  subject 
could  browse  through  this  list  and  select  topics  of 
interest.  The  INDEX  procedure  would  then  present  a  set  of 
locations  in  the  file  where  the  topic  was  located  that  the 
subject  could  scan  quickly.  In  design,  the  INDEX  procedure 
was  modeled  after  a  book  index. 

The  string  search  procedures  (SEARCH,  SEARCH-AND,  and 
SEARCH-AND-NOT)  permitted  the  subjects  to  look  for  specific 
textual  strings  in  the  current  file.  All  string  search 
procedures  would  fill  a  10-location  buffer  which  could  be 
scanned  by  the  subject.  The  SEARCH  procedure  was  a  single 
string  search,  whereas  the  SEARCH-AND/SEARCH-AND-NOT 


procedures  were  multiple  string  searches  with  Boolean 
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operators.  The  SEARCH-AND  procedure  allowed  subjects  to 
look  for  several  textual  strings  within  one  window  (7-lines) 
of  each  other.  The  SEARCH-AND-NOT  procedure  could  be  used 
to  search  for  a  textual  string  (or  strings)  without  the 
occurrence  of  another  textual  string  (or  strings). 

The  fine  adjustment  procedures  moved  the  current  file 
through  the  window  either  one  line  at  a  time  (SCROLL  UP  and 
DOWN)  or  seven  lines  at  a  time  (PAGE  UP  or  DOWN).  Both  the 
scrolling  and  paging  procedures  could  be  used  continuously. 

Procedures  for  information  retrieval .  Data  collection 
for  each  subject  was  conducted  in  6  sessions  over  a  time 
span  of  5  to  7  days.  Individual  sessions  were  scheduled  so 
that  there  was  no  less  than  6  hours  and  no  more  than  48 
hours  between  sessions.  For  the  most  part,  however, 
subjects  scheduled  one  session  on  6  consecutive  days.  The 
extended  time  period  of  the  experiment  was  required  to  train 
and  observe  subjects  learning  to  use  a  software  interface  in 
a  complex  problem-solving  task.  A  summary  of  the  activities 
and  approximate  times  during  these  6  sessions  is  provided  in 
Table  2. 

The  first  two  sessions  shown  in  Table  2  were  devoted  to 
training  and  practice  on  the  information  retrieval  system. 
In  Session  1,  all  subjects  first  received  preliminary 
instructions  on  the  experiment  and  tasks  (Appendix  D).  All 
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TABLE  2 


Experimental  Times  and  Procedures 


Task  Time  (in  minutes) 


Session  1 :  Training 


a.  Introduction  to  experiment  10 

b.  Tutorial  on  database  20 

c.  Break  5 

d.  Tutorial  on  search  procedures: 

SCROLL  to  SEARCH-AND-NOT  55 

e.  Break  5 

f.  Tutorial  on  search  procedures: 

INDEX  to  FILE  SELECT  30 


Total:  125 


Session  2:  Practice 


a.  Reexamination  of  database  and  search 

procedure  reference  sheet  10 

b.  Individual  search  procedure  review  15 

c.  12  practice  trials:  all  search  procedures 

available  30 

d.  Break  5 

e.  12  practice  trials:  training  assessment  20 

f.  Break  5 

g.  Database  recognition  tests  10 


Total 


95 
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TABLE  2 


Continued 


Session  3-5:  Information  Retrieval 

a . 

4  warm-up  trials 

5 

b. 

12  experimental  trials 

20 

c. 

Break 

5 

d. 

12  experimental  trials 

20 

Total:  50 


Session  6:  Information  Retrieval  and  Sorting  Procedures 


a.  4  warm-up  trials  5 

b.  12  experimental  trials  20 

c.  Break  5 

d.  12  experimental  trials  20 

e.  Break  5 

f.  Sorting  of  search  problems  60 


Total 


115 
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of  the  instructions  and  training  given  in  Session  1  were 
administered  interactively  by  the  computer.  Specifically, 
instructions  were  presented  visually  on  a  third  Digital 
VT100,  while  a  Digital  DECtalk  text-to-speech  synthesizer 
repeated  the  instructions  verbally. 

Following  these  introductory  remarks,  all  subjects  were 
introduced  to  the  information  retrieval  database.  An  online 
tutorial  explained  both  the  structure  and  the  content  of  the 
file  hierarchy.  The  tutorial  briefly  summarized  the 
important  aspects  of  each  file  with  regards  to  the  type  of 
information  (e.g.,  verbal  or  numerical)  and  the  file 
structure.  While  reading  and  listening  to  this  tutorial, 
subjects  were  encouraged  to  locate  the  file  on  the  file 
diagram  (Figure  1)  and  to  locate  important  information  in  an 
outlined  and  highlighted  hard  co^  ,  of  the  database.  The 
hard  copy  of  the  database  was  available  only  during  this 
♦•"torial,  while  the  file  diagram  was  available  throughout 
Session  1.  The  text  for  this  online  tutorial  is  provided  in 
Appendix  E. 

After  being  familiarized  with  the  information  retrieval 
database,  subjects  were  asked  to  take  a  5  minute  break. 
Immediately  after  this  break,  all  subjects  were  introduced 
to  the  information  retrieval  system  by  an  online  tutorial. 
The  structure  of  this  tutorial  on  search  procedures  was  to 
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first  explain  both  the  function  and  use  of  each  search 
procedure  separately.  Intermixed  in  these  explanations  were 
actual  information  retrieval  problems  for  subjects  to  solve 
that  demonstrated  small  components  of  each  search  procedure. 
Following  these  explanations  would  be  two  additional  search 
problems  for  the  subject  to  solve  using  only  the  search 
procedures  just  explained.  Both  the  text  and  the  search 
problems  used  in  the  tutorial  are  given  in  Appendix  F. 

At  all  times  the  tutorial  emphasized  that  subjects  should 
see  the  experimenter  when  encountering  difficulty  on  a 
search  procedure.  Furthermore,  repeat  displays  of  the 
tutorial  and  information  retrieval  system  were  monitored  in 
a  separate  room  allowing  the  experimenter  to  determine  if 
there  were  any  major  difficulties  during  training.  Finally, 
all  subjects  had  a  short  search  procedure  reference  sheet 
(see  Appendix  G)  which  briefly  explained  the  use  of  each 
search  procedure.  This  search  procedure  reference  sheet  was 
available  throughout  the  experiment. 

In  Session  2  of  the  experiment,  all  subjects  were  given 
the  opportunity  to  practice  the  search  procedure  skills 
acquired  in  Session  1.  First,  subjects  were  given  a  short 
time  to  reexamine  both  the  database  and  the  search  procedure 
methods.  This  was  followed  immediately  by  a  procedural 
review  of  the  search  procedures.  In  this  review,  subjects 
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were  asked  to  solve  the  same  search  problems  administered  in 
the  Session  1.  After  refreshing  their  memories  on  the 
search  procedures,  subjects  were  given  12  new  search 
problems.  Subjects  were  encouraged  during  these  trials  to 
explore  the  different  search  procedures.  During  these 
trials,  the  experimenter  was  available  for  questions. 
Moreover,  if  the  subject  did  not  interact  with  the 
information  retrieval  system  for  more  than  2  minutes,  the 
experimenter  entered  the  experimental  room  and  asked  whether 
the  subject  had  any  questions. 

Following  the  initial  whole  training  (all  search 
procedures  available)  subjects  were  given  a  5  minute  break, 
followed  by  an  additional  12  search  problems.  Before 
beginning  these  12  search  problems,  subjects  were  instructed 
that  the  experimenter  would  not  be  available  for  questions 
and  that  they  were  to  retrieve  the  information  as  fast  as 
possible.  Based  on  pretesting,  an  average  of  300  seconds 
was  set  as  the  upper  time  limit  for  all  subjects  during 
these  trials.  Both  this  performance  test  and  the  subsequent 
test  on  the  database  were  required  to  insure  that  all 
subjects  had  basic  information  retrieval  skills. 

As  shown  in  Table  2,  all  subsequent  sessions  (3  -  6) 
involved  information  retrieval.  Subjects  during  these 
sessions  received  5  warm-up  trials,  followed  by  24  search 
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problems  that  were  split  into  2,  12-trial  blocks  separated 
by  a  5-minute  break.  Consequently,  subjects  received  96 
search  problems  (Appendix  H)  on  which  performance  was 
analyzed.  On  these  96  trials  subjects  were  asked  to  find 
the  information  as  fast  as  possible. 

The  presentation  order  of  the  96  search  problems  was 
randomized  with  two  restrictions.  The  first  restriction 
required  that  presentation  order  be  matched  for  novices  and 
experts.  This  prevented  any  presentation  orders 
differentially  biasing  the  performance  of  either  subject 
group.  The  second  restriction  required  that  a  set  of  search 
problems  be  randomly  selected  for  Session  6  and  the 
presentation  order  be  randomized  separately  from  the  other 
sessions.  This  restriction  was  required  to  later  compare 
transfer  performance  of  the  advised  subjects  in  Experiment  2 
with  the  performance  of  subjects  in  Experiment  1. 

Procedures  for  sorting  search  problems.  As  illustrated 
by  Table  2.  L'.e  key  differ eitc*  between  Sessions  3-5  and 
Session  6  was  the  presence  of  the  sorting  procedures  at  the 
end  of  Session  6.  In  these  procedures,  both  novice  and 
expert  subjects  were  asked  to  sort  the  96  search  problems 
into  meaningful  categories.  The  sorting  procedure  was  a 
variant  of  the  two-level  sorting  method  used  by  Tullis 
(1985)  to  structure  a  menu  operating  system. 
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To  sort  the  search  problems,  subjects  were  seated  at  a 
desk  with  the  96  search  problems  typed  on  index  cards. 
Subjects  then  were  given  written  instructions  (see  Appendix 
I)  on  how  to  sort  the  search  problems.  During  the  first 
sorting  phase,  subjects  were  instructed  to  place  the  search 
problems  into  meaningful  groups.  No  restrictions  were 
placed  on  the  number  of  groups  or  number  of  search  problems 
in  each  group.  However,  subjects  were  told  to  supply  on  a 
separate  index  card:  (1)  a  list  of  group  characteristics  or 
a  reason  why  the  search  problems  were  classified  together, 
and  (2)  a  possible  sequence  of  search  procedures  (a  search 
plan)  for  locating  that  type  of  search  problem.  Subjects 
were  allowed  as  much  time  as  possible  to  complete  the  first 
sorting  phase,  however,  most  subjects  finished  in  under  60 
minutes . 

In  the  second  phase  of  sorting,  subjects  were  to  perform 
tne  sorting  process  once  more.  That  is,  subjects  were  asked 
to  sort  the  groups  of  search  problems  generated  in  the  first 
sort  into  more  general  categories  and  to  provide  a  short 
description  of  these  more  general  categories.  Subjects  were 
not  allowed  to  restructure  the  groups  from  the  first  phase 
and  were  given  as  much  time  as  necessary  to  complete  the 
second  sort.  Most  subjects  finished  the  second  sorting 
phase  in  approximately  10  minutes. 
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As  a  result  of  these  two  sorting  phases,  subjects  formed 
a  two- level  hierarchy  of  search  problems  that  could  be 
scored  using  a  procedure  described  by  Tullis  (1985). 
Specifically,  a  96  X  96  distance  matrix  was  constructed  by 
adding  a  0,  1,  or  2  to  each  cell  indicating  the  similarity 
of  two  search  problems  for  each  subject.  A  0  was  added  to 
the  appropriate  cell  if  the  two  search  problems  appeared  in 
the  same  group  at  the  end  of  the  first  sorting  phase.  A  1 
was  added  to  the  appropriate  cell  if  the  two  search  problems 
were  not  in  the  same  group  after  the  first  sorting  phase, 
but  appeared  together  in  a  more  general  group  after  the 
second  phase  of  the  sorting  procedure.  Finally,  a  2  was 
added  to  the  appropriate  cell  if  the  search  problems 
appeared  in  no  group  together.  In  other  words,  the  0,  1,  or 
2  was  a  simple  metric  for  describing  group  membership  of  the 
search  problems  in  the  two-level  hierarchy  for  each  subject. 

A  two-level  sort  was  selected  for  two  reasons.  Most 
importantly,  the  two-level  sort  generated  a  hierarchical 
structure  that  was  consistent  with  the  hierarchical 
database.  Second,  a  two-level  sort  allowed  subjects  more 
than  one  chance  to  use  different  criteria  in  sorting  the 
search  problems  (Rosenberg  and  Kim,  1975).  Clearly,  there 
were  several  dimensions  on  which  search  problems  could  be 
classified.  Subjects,  for  example,  could  sort  search 
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problems  based  on  the  type  of  search  strategy  required,  the 
location  in  the  database,  or  the  type  of  numerical  or  verbal 
information  requested.  A  two-level  sort  made  it  possible  to 
identify  and  structure  these  dimensions. 

Results  and  Discussion 

Search  performance .  Search  performance  was  analyzed  for 
the  total  number  of  search  operations,  number  of  different 
search  operations  used,  total  time  in  seconds,  and  total 
file  movement  through  the  display  window  corrected  for  the 
minimum  movement  to  a  search  problem  (in  lines).  A 
multivariate  analysis  of  the  novice  and  expert  means  for 
these  dependent  measures  indicated  that  there  were 
statistically  reliable  differences  in  overall  search 
performance,  Hotelling's  T2  =  15.94,  F(4,19)  =  3.45, 
g  =  0.0281.  Box  plots  of  the  means  for  the  four  dependent 
variables  are  presented  in  Figure  3.  Represented  in  each  of 
these  box  plots  are  the  interquartile  range  (the  outer  edges 
of  each  box),  the  median  (the  middle  line  in  each  box),  the 
mean  (the  asterisk  in  each  box),  and  the  data  points  falling 
outside  the  interquartile  range.  Figure  3  shows  the 
differences  in  mean  search  performance  between  novices  and 
experts  and  also  the  differences  in  the  search  variability 
of  the  two  groups.  That  is,  novices  were  slower  and  more 
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variable  in  locating  information  in  the  database  than  exnert 
subj  ects . 

Separate  two-way,  analyses  of  variance  (ANOVAs)  were 
conducted  on  these  dependent  measures  to  determine  if  there 
were  any  mean  search  differences  that  could  be  detected  for 
novices  and  experts  across  the  four  retrieval  sessions. 
Summaries  of  these  and  all  other  ANOVAs  are  contained  in 
Appendix  J.  The  results  of  the  four  ANOVAs  revealed 
statistically  reliable  differences  between  novices  and 
experts  for  total  operations  (£[1,22]  =  6.04,  jj  =  0.0224) 
and  total  time  (£[1,22]  =  8.59,  E  =  0.0077).  From  these 
analyses  and  the  means  plotted  in  Figure  3,  novices  appear 
to  take  more  time  ana  search  operations  to  find  information 
when  compared  to  experienced  subjects. 

The  four  ANOVAs  on  search  performance  also  revealed 
highly  significant  session  effects  for  total  operations 
(£[3,66]  =  11.07,  p  =  0.0001),  different  operations 
(£[3,66]  =  6  42,  e  =  0.0008),  total  time  (£[3,66]  =  20.73, 
E  =  0.0001),  and  total  movement  (£[3,66]  =  4.91,  e  =  0.004). 
These  session  effects  indicate  that  subjects  were  learning 
how  to  retrieve  information  in  this  complex  task.  However, 
the  lack  of  any  significant  session  by  subje  :  experience 
interaction  (e  >  0.05)  indicates  that  both  novices  and 
experts  were  learning  at  the  same  rate. 


Unfortunately,  ma:  .y  of  the  above  results  are  suspect  due 
to  the  large  differences  in  variability  between  novices  and 
experts.  To  analyze  search  performance  variation  of  novice 
and  expert  subjects,  a  non-parametric,  Moses  rank-like  test 
(Hollander  and  Wolfe,  1973)  was  conducted.  A  nonparametric 
test  was  chosen  to  be  conservative  in  making  assumptions 
about  the  underlying  search  distributions.  As  a  result  of 
these  tests,  significant  differences  in  search  performance 
variability  were  found  for  total  operations  (W[4,4)  =  25.0, 
p  =  0.029),  and  total  time  (W[4,4J  =  26.0,  p  =  0.014).  The 
magnitude  of  these  differences  were  enormous  both  from 
observation  of  Figure  3  and  the  variances  of  t^tal 
operations  (novices,  a 2  =  19.42;  experts,  o2  =  2.54)  and 
total  time  (novices,  o2  =  869.28;  experts,  a2  =  80.42). 

Further  inspection  of  search  performance  in  Figure  3 
provides  additional  insights  into  the  large  differences  in 
variability  among  novices.  Tor  instance,  if  all  novice 
search  distributions  are  halved  at  the  median,  comparisons 
of  the  fast  and  efficient  novices  with  the  experts  would 
yield  no  differences.  Comparisons  of  the  slow  and 
inefficient  novices  with  the  experts,  however,  would  lead  to 
large  differences.  Simply  put,  two  different  types  of 
novice  users  may  have  been  sampled  in  this  experiment. 
Unfortunately,  the  lack  of  any  significant  correlation 


(Spearman's  rank  correlation  coefficient,  p  <  0.05)  between 
background  variables  and  any  search  performance  measure 
prevented  the  formation  of  data-driven  hypotheses  on  the 
underlying  characteristics  that  separate  novice  users. 

Nevertheless,  examination  of  the  search  performance  for 
"slow"  novices,  "fast"  novices,  and  experts  over  the 
experimental  sessions  were  useful  to  identify  differences  in 
the  learning  curves  that  could  affect  the  implementation  of 
the  online  aid.  To  conduct  these  analyses,  novice  subjects 
were  divided  equally  into  a  slow  group  (6  subjects)  and  a 
fast  group  (6  subjects)  based  on  average  time.  Four 
separate  ANOVAs  with  unequal  cell  sizes  were  employed  for 
each  dependent  measure  using  the  method  of  weighted  means 
(Winer,  1971 ) . 

Not  surprisingly,  highly  significant  subject  group 
effects  were  found  for  total  operations  (£[2,21]  =  14.26, 
p  =  0.0001),  different  operations  (£[2,21]  =  7.50, 
p  =  0.0035),  total  time  (£[2,21]  =  27.86,  p  =  0.0001),  and 
total  movement  (£[2,21]  =  8  61,  p  =  0.0019).  In  addition, 
much  like  the  previous  balanced  ANOVAs,  highly  significant 
session  effects  were  found  for  total  operations 
(£[3,63]  =  11.24,  P  =  0.0001),  different  operations 
(£[3,63]  =  6.14,  p  =  0.0011),  total  time  (£[3,63]  =  21.17, 
E  =  0.0001),  and  total  movement  (£[3,63]  =  4.32,  p  =  0.0079) 
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without  any  significant  (p  <  0.05)  subject  group  by  session 
interactions.  All  of  these  results  are  summarized  in  Figure 
4  which  displays  the  average  search  measures  for  all  three 
subject  groups  (slow  novices,  fast  novices,  and  experts) 
over  all  sessions.  Surprisingly,  Figure  4  shows  how  closely 
fast  novices  resembled  experts  in  search  performance 
throughout  the  experiment.  These  two  subject  groups  (fast 
novices  and  experts)  were  significantly  faster  than  slow 
novices  at  all  phases  of  the  experiment  (Newman-Keuls, 
a  =  0.05) . 

The  results  of  these  analyses  on  search  performance 
demonstrated  that  the  differences  between  novices  and 
experts  existed  both  in  terms  of  average  search  performance 
(total  operations  and  time)  and  the  variability  in  search 
performance.  However,  both  the  average  and  variability 
differences  are  closely  related  since  the  slower  novices 
strongly  influenced  the  overall  mean  search  performance  of 
the  group.  As  a  result,  the  goal  of  online  aiding  with  this 
information  retrieval  task  should  be  to  assist  and  instruct 
those  novice  users  having  difficulties  locating  information 
without  interfering  with  novices  that  are  performing 
adequately.  The  implications  for  this  type  of  online  aid 
can  be  viewed  from  both  an  individual  and  a  group 
perspective.  For  specific  individuals,  online  aiding  may 


TOTAL  TIME  (SECONDS)  TOTAL  OPERATIONS 
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LEGEND:  GROUP 


k - k - k  EXPERTS 

0 - O - 0  fast  NOVICES 

# - * - *  SLOW  NOVICES 


Figure  4.  Search  performance  for  fast  novices,  slow  novices, 
and  experts  over  sessions .  I 
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improve  their  performance  so  that  the  automated  system  can 
be  used  for  real  tasks.  From  a  group  perspective,  online 
aiding  may  yield  a  more  homogeneous  group  of  users  that 
would  be  easier  to  support  and/or  train.  Both  of  these 
goals  are  important  and  have  been  stated  in  other  research 
areas  such  as  computer-assisted  instruction  (Atkinson, 
1972) . 

The  search  performance  results  were  also  interesting 
since  the  novice-expert  differences  resembled  some  of  the 
empirical  data  on  acquiring  high-performance  skills 
(Schneider,  1985).  For  example,  the  high  degree  of 
heterogeneity  in  novice  search  performance  has  been  found 
frequently  when  training  other  high-performance  skills. 
Thus,  the  heterogeneity  of  novice  search  performance  may  not 
just  be  a  statistical  artifact  that  should  have  been 
controlled,  but  a  real  phenomenon  resulting  from  a  highly 
structured  and  complicated  task. 

Search  strategy.  Selection  of  search  procedures  by 
novice  and  expert  subjects  was  analyzed  based  on  a  polling 
procedure  developed  by  Elkerton  and  Williges  (1985).  A 
diagram  of  this  polling  procedure  for  a  hypothetical  subject 
is  presented  in  Table  3.  Conceptually,  a  subject  on  each 
trial  was  given  a  vector  of  12  votes  representing  the  12 
search  procedures.  A  vote  was  cast  for  a  search  procedure 


TABLE  3 


Polling  Pro 


SEARCH 
PROCEDURES  i 

SCROLL  UP 
SCROLL  DOWN 
PAGE  UP 
PAGE  DOWN 
SECTION 
SEARCH 
SEARCH- AND 
SEARCH-NOT 
INDEX 
ZOOM-IN 
ZOOM-OUT 
FILE  SELECT 
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when  the  subject  used  the  search  procedure  at  least  once 
during  the  trial.  The  votes  were  summed  across  the  96 
trials  to  yield  the  total  polls  for  each  search  procedure 
and  converted  into  a  proportion  indicating  the  relative  use 
of  a  search  procedure  for  each  subject.  The  polling 
procedure  was  used  to  protect  against  the  possibility  of  a 
highly  repeated  search  procedure  (e.g.,  SCROLL  UP,  SCROLL 
DOWN,  PAGE  UP,  and  PAGE  DOWN)  biasing  the  strategy  analysis 
on  one  trial. 

The  12  search  procedure  proportions  for  novice  and  expert 
subjects  were  analyzed  with  a  multivariate  analysis  of 
variance  (MANOVA)  to  determine  if  there  any  overall  search 
strategy  differences.  A  MANOVA  model  was  chosen  to  account 
for  the  ass  .elation  in  the  selection  of  individual  search 
procedures.  The  results  of  the  one-way  MANOVA  indicated 
there  were  no  differences  in  the  selection  of  search 
procedures  by  novices  and  experts,  Wilks'  A(l,22)  =  0.2741, 
p  =  0.0723.  This  strategy  analysis  then  was  supplemented 
with  a  MANOVA  on  search  procedure  proportions  for  the  slow 
novices  (6  subjects),  fast  novices  (6  subjects),  and  experts 
(12  subjects)  since  previous  analyses  revealed  heterogeneity 
of  novice  search  performance.  The  results  of  this 
additional  strategy  analysis  indicated  that  there  were 
differences  in  the  selection  of  search  procedures  by  slow 
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novices,  fast  novices,  and  experts,  Wilks'  A(2,21)  =  0.0342, 

p  =  0.0022. 

To  further  investigate  the  differences  in  search 
strategies,  12  unbalanced  ANOVAs  on  individual  search 
procedure  proportions  were  conducted.  The  results  of  these 
ANOVAs  revealed  a  significant  difference  in  the  selection  of 
SCROLL  UP  (F[ 2, 21]  =4.57,  p  =  0.0225),  SCROLL  DOWN 
( F [ 2 , 21 ]  =  5.85,  p  =  0.0096),  and  ZOOM  OUT  (F[2,21]  =  11.00, 
p  =  0.0005),  by  slow  novices,  fast  novices,  and  experts. 
Figure  5  shows  the  proportion  of  polls  and  results  of 
Newman-Keuls  procedures  (a  =  0.05)  for  slow  novices,  fast 
novices,  and  experts  on  these  three  search  procedures. 

As  illustrated  in  Figure  5,  slow  novices  selected  SCROLL 
UP,  SCROLL  DOWN,  and  ZOOM  OUT  procedures  more  frequently 
than  fast  novices  and  experts.  The  increased  use  of 
scrolling  procedures  by  slow,  inexperienced  subjects 
replicates  the  results  of  Elkerton  and  Williges  (1984a, 
1985)  and  indicates  that  slow  novices  may  benefit  from 
search  procedure  suggestions  provided  by  an  online  aid.  To 
use  Schneider's  (1985)  recommendation,  poor  or  inefficient 
strategies  should  be  eliminated  when  training  a  complex 
skill  like  information  retrieval.  Secondly,  the  increased 
use  of  the  ZOOM  OUT  procedure  could  be  an  indication  that 
slower  subjects  more  frequently  selected  the  wrong  file  and 


a: 

a. 


ZOOM  OUT 


SCROLL  UP  SCROLL  DOWN 

LEGEND:  SLOW  NOVICES  -  □  FAST  NOVICES  -  ■ 

EXPERTS  -  □ 


Figure  5.  Significant  differences  in  the  selection  of  search 
procedures  for  slow  novices,  fast  novices,  and  experts ■ 
Groups  with  a  common  horizontal  line  are  not  significantly 
different,  Newman-Keuls  (a  =  0.05). 
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had  to  ZOOM  OUT  to  further  search  the  file  hierarchy.  An 
online  aid  could  prevent  this  by  providing  clues  on  the 
general  location  of  a  search  problem  in  the  file  hierarchy. 

Search  clusters.  To  provide  intelligent  advice  on  the 
selection  of  search  procedures  requires  that  the  task 
materials  (search  problems)  be  organized  into  a  meaningful 
structure  for  users.  Without  this  structure, 
command-selection  aids  could  only  offer  very  general  advice 
(what  search  procedure  to  use  based  on  all  trials)  or  very 
specific  advice  (what  search  procedure  to  use  for  a 
particular  trial).  To  extract  the  task  structure,  the 
96  X  96  distance  matrix  for  expert  subjects  was  analyzed 
with  hierarchical  cluster  analysis  using  the  centroid  method 
(Everitt,  1974).  Expert  subjects  were  focused  on  during 
cluster  analysis  since  the  performance  analyses  found  these 
subjects  to  be  a  more  homogeneous  group  than  novices. 

The  hierarchical  cluster  analysis  was  used  as  a  graphical 
technique  to  suggest  homogeneous  sets  of  search  problems. 
During  this  analysis,  the  information  in  the  search  problems 
and  the  file  and  line  locations  of  the  search  problems  were 
used  to  develop  the  search  clusters.  The  cluster  analysis 
was  started  at  15  clusters.  Although  this  starting  point 
was  somewhat  arbitrary,  a  moderate  number  of  search  clusters 
was  desired  since  an  extreme  number  would  impede 
construction  of  the  command-selection  models. 
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At  15  clusters  there  were  two  search  problems  (Items  34 
and  43  in  Appendix  H)  which  formed  two  individual  clusters. 
To  avoid  these  single  item  clusters  the  analysis  proceeded 
to  13  clusters  where  one  of  the  search  problems  (Item  43) 
joined  a  cluster  and  the  other  (Item  34)  remained  as  a 
single  cluster.  At  this  level,  several  reasonable  search 
clusters  were  clear  and  are  described  in  Table  4.  Included 
in  Table  4  are  examples  of  representative  search  problems. 
In  addition  to  Table  4,  the  dendogram  in  Figure  6  shows  the 
hierarchical  structure  of  the  12  multiple  item  clusters  and 
the  size  of  these  search  clusters. 

A  decision  was  made  to  stay  at  12  clusters  since  moving 
up  the  dendogram  would  immediately  join  Clusters  7  and  8 
(~ee  Figure  6).  Clusters  7  and  8  were  not  joined  since 
these  search  problems  contained  specific  information  that 
identified  the  tank  or  APC  (armored  personnel  carrier) 
equipment  file  where  the  information  could  be  located. 
Joining  these  two  sets  of  search  problems  would  destroy  the 
file-location  specificity  which  was  suggested  as  a  desirable 
feature  of  an  online  aid  in  the  search  strategy  analyses. 

The  problem  of  Item  34  being  a  singular  cluster  was 
resolved  by  inspection.  Item  34  (Locate  the  vehicle  with  ID 
code  34037  that  is  on  reserve  and  has  205  battle  hours)  was 
a  difficult  search  problem  that  relied  on  subtle  memory 
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TABLE  4 


Descriptions  and  Examples  of  the  12  Search  Clusters 


Cluster  1  -  Specific  APC  Inventory:  Locate  the  BTR-60 

armored  personnel  carrier  with  ID  code  4AAE3  that  is  in 
the  Forest  regiment. 

Cluster  2  -  Specific  Tank  Inventory:  Locate  the  PT-76  tank 
on  reserve  with  the  most  battle  hours. 

Cluster  3  -  Unspecified  APC  Inventory  (Embedded):  Locate  the 
maintenance  battalion  of  the  armored  carrier  with  an  ID 
code  of  69024. 

Cluster  4  -  Unspecified  Tank  Inventory  (Embedded) :  Locate 
the  battle  tank  with  ID  code  21XF. 

Cluster  5  -  Unspecified  APC  Inventory  (Nonembedded) :  Locate 
the  total  number  of  armored  personnel  carriers  in 
regiment  number  499. 

Cluster  6  -  Unspecified  Tank  Inventory  (Nonembedded):  Locate 
the  total  number  of  battle  tanks  in  the  Renegade 
regiment . 

Cluster  7  -  Equipment  for  APC:  Locate  the  height  of  the 

BMP-1  armored  personnel  carrier. 

Cluster  8  -  Equipment  for  Tank:  Locate  the  method  of  ranging 
for  the  T-10  tank. 

Cluster  9  -  Combat  Support  Information  (Subgroups):  Locate 
the  only  subgroup  of  the  anti-aircraft  gun  regiment. 

Cluster  10  -  Combat  Support  Information  (Mission  and 

Operations):  Verify  that  the  engineer  battalion  can 

operate  a  sawmill. 

Cluster  11  -  Army  Operations  (Numerical):  Locate  the  number 
of  enlisted  personnel  in  the  medical  battalion. 

Cluster  12  -  Army  Operations  (Verbal):  Verify  that  the  tank 
division  will  conduct  mobile  counterattack  operations. 
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NUMBER  OF  CLUSTERS 

CLUSTER  DESCRIPTION  121112882851321 


1.  SPECIFIC  APC  INVENTORY 
(17  TARGETS) 


2.  SPECIFIC  TANK  INVENTORY _ 

(14  TARGETS) 

3.  UNSPECIFIED  APC  INVENTORY: 
EMBEDDED  (8  TARGETS) 


4.  UNSPECIFIED  TANK  INVENTORY: 
EMBEDDED  (9  TARGETS) 


5.  UNSPECIFIED  APC  INVENTORY: 
NONEMBEDDED  (4  TARGETS) 


6.  UNSPECIFIED  TANK  INVENTORY: 
NONEMBEDDED  (3  TARGETS) 

7.  EQUIPMENT  FOR  APC  _ _ 

(10  TARGETS) 


8.  EQUIPMENT  FOR  TANK _ 

(10  TARGETS) 

9.  COMBAT  SUPPORT  INFORMATION: 
SUBGROUPS  (7  TARGETS) 


10.  COMBAT  SUPPORT  INFORMATION: 
MISSION  AND  OPERATIONS 
(7  TARGETS) 


11.  ARMY  OPERATIONS: 
NUMERICAL  (3  TARGETS) 

12.  ARMY  OPERATIONS: 
VERBAL  (4  TARGETS) 


Figure  6.  Dendogram  of  the  12  search  clusters . 
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cues.  Subjects  would  have  to  recall  that  ID  codes  with  5 
places  denoted  an  armored  personnel  carrier  located  in  an 
inventory  file.  As  a  result.  Item  34  was  placed  in  the 
cluster  associated  with  an  unspecified  and  embedded  armored 
personnel  carrier  inventory  (Cluster  3).  A  similar  memory 
problem  was  clear  with  Item  9  (Locate  the  number  of  vehicles 
in  the  Razor  regiment).  Originally,  the  cluster  analysis 
placed  this  search  problem  in  the  cluster  related  to 
unspecified  and  nonembedded  tank  information  (Cluster  6). 
However,  the  actual  location  of  the  search  problem  was  at 
the  top  of  one  of  the  armored  personnel  carrier  inventory 
files.  The  search  problem,  therefore,  was  shifted  to 
Cluster  5  (Unspecified  APC  Inventory:  Nonembedded). 

Aside  from  these  slight  memorv- related  anomalies,  the 
results  of  the  sorting  and  cluster  analysis  were  satisfying 
since  the  structure  of  the  search  clusters  resembled  the 
structure  of  the  database  (see  Figure  1).  For  example,  the 
dendogram  presented  in  Figure  6  illustrates  that  experienced 
subjects  sorted  search  problems  into  search  clusters 
associated  with  inventory  information  (Clusters  1  -  6), 
search  problems  associated  with  equipment  (Clusters  7  and 
8),  combat  support  search  problems  (Clusters  9  and  10),  and 
search  problems  related  to  army  operations  (Clusters  11  and 
12). 
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In  addition,  several  other  dimensions  of  the  search 
problems  emerged  from  the  dendogram.  Experts  subjects  in 
sorting  the  search  problems  made  a  clear  distinction  between 
specific  inventory  problems  (Clusters  1  and  2)  and 
unspecified  inventory  problems  (Clusters  3-6).  The  latter 
class  of  search  problems  required  subjects  to  search  several 
inventory  files.  Further  distinctions  were  made  within  the 
unspecified  inventory  problems  with  regard  to  the 
approximate  location  of  the  item  in  the  inventory  file.  In 
all  inventory  files  a  summary  table  was  available  at  the  top 
of  the  file.  Search  problems  located  in  the  summary  table 
were  clustered  together  and  were  classified  as  nonembedded 
items  (Clusters  5  and  6).  In  contrast,  embedded  search 
problems  (Clusters  3  and  4)  were  located  throughout  the 
inventory  files  and  frequently  required  a  ballistic  search 
movement  into  the  body  of  the  file  for  item  location. 
External  empirical  support  for  the  validity  cf  the 
embedded-nor  ambedded  dimension  can  be  found  in  an  earlier 
file  search  investigation  by  Elkerton  and  Wiiliges  (1984a). 
These  investigators  found  significant  increases  in  retrieval 
time  and  effort  for  embedded  search  items  when  compared  to 
nonembedded  items. 

The  last  twc  dimensions  evident  in  the  dendogram  of 
Figure  6  were  concerned  with  the  actual  content  of  the 
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combat  support  files  and  the  army  operations  file. 
Information  contained  in  the  combat  support  files  appeared 
to  be  separated  with  regard  to  subgroups  (Cluster  9)  and 
mission  or  operational  capabilities  (Cluster  10).  The 
source  of  this  distinction  was  clear  when  the  presentation 
formats  of  subgroups  and  mission  or  operational  capabilities 
were  examined.  Subgroup  data  in  the  combat  support  files 
were  available  in  column  lists,  while  mission  or  operational 
capabilities  were  presented  as  sentence  lists.  A  similar 
distinction  was  drawn  in  the  army  operations  file.  Expert 
subjects  appeared  to  separate  search  problems  based  on 
whether  information  was  numerical  and  column  formatted 
(Cluster  11),  or  verbal  and  sentence  formatted  (Cluster  17). 

The  12-cluster  solution  presented  in  Figure  6  represents 
a  model  based  on  the  sorting  behavior  of  all  expert 
subjects.  This  overall  model  was  thought  to  be  more 
appropriate  than  a  hierarchy  generated  by  a  single  expert 
due  to  the  large  variability  in  sorting  behavior.  The 
number  of  lower-level  search  groups  generated  by  experts 
after  the  first  sorting  phase  ranged  from  3  to  33  groups 
with  a  median  of  9  groups.  In  the  second  sorting  phase, 
experts  sorted  the  lower-level  search  groups  into  2  to  9 
higher-level  groups  with  a  median  of  4  groups.  This 
variability  indicated  that  an  individual  solution  had  the 
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potential  of  being  an  idiosyncratic  classification  of  the 
search  problems. 

Empirical  support  for  the  12  search  clusters  can  be  found 
both  in  the  search  performance  and  search  strategy  data. 
However,  analysis  of  this  data  was  not  straightforward  since 
the  search  clusters  were  determined  after  collecting  the 
information  retrieval  data.  As  a  result,  the  number  of 
items  in  each  search  cluster  could  not  be  controlled.  To 
overcome  this  limitation,  the  means  of  the  dependent 
measures  for  each  search  cluster  and  subject  were  computed 
and  analyzed  in  a  balanced  ANOVA. 

Two-way  ANOVAs  were  conducted  to  determine  the  effect  of 
search  cluster  on  search  performance  for  both  expert  and 
novice  subjects.  The  results  of  the  ANOVAs  revealed  highly 
significant  effects  of  search  clusters  for  total  operations 
(F[ll,242]  =  11.72,  p  =  0.0001),  different  operations 
(F[ 11 , 242 ]  =  16.31,  p  =  0.0001),  total  time 
(F[ll,242]  =  5.65,  p  =  0.0001),  and  total  movement 
(F[ll, 242]  =  11.06,  p  =  0.0001).  Correlation  ratios,  n2, 
(Thorndike,  1978)  were  calculated  and  indicated  that  search 
cluster  could  describe  24.1%,  29.2%,  14.4%,  and  25.3%  of  the 
search  variance  for  the  averages  of  total  operations, 
different  operations,  total  time,  and  total  movement, 
respectively.  These  search  cluster  effects  demonstrate  that 
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the  type  of  target  influenced  a  subject's  search 
performance.  Moreover,  the  lack  of  any  significant 
(p  <  0.05)  subject  experience  by  search  cluster  interaction 
indicates  that  the  performance  of  novice  and  expert  subjects 
was  not  influenced  differentially  by  search  clusters. 
Further  unbalanced  ANOVAs  that  split  novices  into  fast  and 
slow  groups  yielded  similar  conclusions. 

Table  5  displays  the  search  performance  means  for  each 
search  cluster  in  decreasing  order.  Included  in  Table  5  are 
the  results  of  a  Newman-Keuls  analysis  (a  =  0.05)  on  the 
means  of  the  12  search  clusters  for  each  search  performance 
measure.  Inspection  of  the  results  displayed  in  Table  5 
suggests  that  the  search  problems  in  Clusters  3,  5,  and  6 
were  consistently  more  difficult  to  find.  These  search 
problems  were  unspecified  inventory  items  that  required  a 
large  amount  of  file  hierarchy  traversal.  Table  5  also 
shows  that  Clusters  1,  2,  7,  8,  and  11  were  found 
consistently  faster  and  more  efficiently  than  other  types  of 
search  problems.  These  items  usually  contained  numerical 
and  statistical  data  in  specific  files.  The  results  in 
Table  5  suggest  that  the  search  clusters  were  not  highly 
differentiated  in  terms  of  search  performance.  However,  the 
lack  of  differentiation  was  expected.  Search  operations, 
time,  and  movement  are  important  file  search  variables,  bur 


75 


TABLE  5 


Effects  of  Search  Clusters  on  Average  Search  Performance 


Total 

Operations 

Different 

Operations 

Total 

Time 

Total 

Movement 

Cluster 

Cluster 

Cluster 

Cluster 

Mean 

# 

Mean 

# 

Mean 

# 

Mean 

# 

17.8 

6 

5.2 

6 

117.88 

6 

277.4 

5 

17.4 

5 

4.8 

5 

114.73 

3 

201.8 

6 

15.0 

3 

4.8 

3 

108.30 

9 

194.  1 

3 

10.5 

4 

4.0 

9 

108.13 

5 

121.0 

4 

9.8 

12 

4.0 

10 

106.64 

12 

101.5 

9 

9.8 

9 

3.9 

4 

88.64 

10 

95.5 

11 

9.1 

10 

3.7 

7 

81.23 

11 

92.3 

12 

8.3 

7 

3.5 

12 

73.72 

4 

88.6 

10 

7.4 

11 

3.4 

8 

72.76 

7 

76.5 

7 

6.8 

8 

3.2 

1 

62.24 

1 

56.4 

1 

5.8 

1 

3.1 

11 

54.71 

8 

45.1 

2 

5.0 

2 

3.1 

2 

48.81 

2 

35.0 

8 

Cluster 

1 

Cluster 

2 

Cluster 

3 

Cluster 

4 

Cluster 

5 

Cluster 

6 

Cluster 

7 

Cluster 

8 

Cluster 

9 

Cluster 

10 

Cluster 

11: 

Cluster 

12  : 

Specific  APC  Inventory. 

Specific  Tank  Inventory. 

Unspecified  APC  Inventory  (Embedded) . 
Unspecified  Tank  Inventory  (Embedded). 
Unspecified  APC  Inventory  (Nonembedded) . 
Unspecified  Tank  Inventory  (Nonembedded) . 
Equipment  for  APC. 

Equipment  for  Tank. 

Combat  Support  Information  (Subgroups). 
Combat  Support  Information  (Mission  and 
Operations ) . 

Army  Operations  (Numerical). 

Army  Operations  (Verbal). 


Kw-ns  with  a  common  vertical  line  are  not  significantly 
different,  Newman-Keuls  (a  =  0.05). 
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are  only  surface  indicators  of  the  cognitive  processes 
required  for  information  retrieval. 

Support  for  the  search  clusters  also  was  found  when 
analyzing  search  strategy  for  different  types  of  search 
problems.  A  three-way,  mixed-factor  ANOVA  was  conducted  on 
subject  experience,  search  procedure,  and  search  cluster  to 
determine  whether  novice  and  expert  subjects  used  different 
search  strategies  for  the  search  clusters.  Subject 
experience  was  the  only  between-subjects  variable  in  this 
ANOVA  with  search  cluster  and  search  procedure  serving  as 
within- subjects  variables.  As  in  previous  search  strategy 
analyses,  the  proportion  of  search  procedure  use  was 
assessed  with  a  polling  procedure  to  prevent  the  possible 
bias  of  a  highly  repeated  search  procedure  on  a  single 
trial . 

Three  statistical  interactions  were  of  interest  in  the 
three-wa  ’  ANOVA.  The  search  procedure  by  subject  experience 
interaction  was  tested  to  check  the  MANOVA  results  on  search 
strategy  for  experts  and  novices.  Consistent  with  the 
conclusions  of  the  MANOVA,  expert  and  novice  subjects  did 
not  select  different  search  procedures  overall 
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search  procedure  by  subject  experience  (F[121,2662]  =  1.58, 
p  =  0.0001)  suggest  that  search  cluster  influenced  the 
selection  of  search  procedures  by  expert  and  novice 
subjects . 

Although  the  large  number  of  degrees  of  freedom  in  these 
tests  resulted  in  a  sensitive  analysis,  these  results  were 
informative  for  two  reasons.  First,  search  clusters  were 
identified  as  an  important  determiner  of  search  strategies. 
Second,  the  search  strategies  of  expert  and  novice  subjects 
were  found  to  differ  when  search  cluster  was  considered. 
The  influence  of  both  interactions  can  be  seen  in  the  search 
strategies  of  novice  and  expert  subjects  displayed  in  Figure 
7  for  Cluster  4  (Unspecified  Tank  Inventory:  Embedded)  and 
Cluster  9  (Combat  Support  Information:  Subgroups).  Included 
in  each  histogram  are  asterisks  which  indicated  whether  a 
Fisher's  LSD  (least  significant  difference)  test  was 
significant  (a  =  0.05)  when  comparing  novice  and  expert 
search  proportions  on  a  specific  search  cluster.  Fisher's 
LSD  test  was  used  only  to  suggest  search  strategy 
differences  in  this  complex  three-way  interaction. 

Examination  of  Figure  7  revealed  large  differences  in  the 
selection  of  the  search  procedures  between  the  two  search 
clusters.  The  profiles  of  the  two  histograms  are  markedly 
different.  Cluster  4,  an  unspecified  and  embedded  tank 
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Figure  7.  Search  strategies  of  novice  and  expert  subjects 
for  Clusters  4  and  9.  Asteri  sks  indicate  si  gni.f  i  c  ant 
differences;""  Fisher's  LSD  (a_  =  0.05),  between  the  selection 
proportions  of  experts  and  novices . 
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inventory  cluster,  required  subjects  to  search  the  tank 
inventory  files  systematically.  Experts  in  Cluster  4 
selected  the  SEARCH  command  more  frequently  than  novices, 
while  novices  used  the  SCROLL  DOWN  procedure  to  a  greater 
extent.  In  contrast  to  Cluster  4,  Cluster  9  required 
subjects  to  search  the  two  combat  support  files  for  subgroup 
information.  In  searching  these  two  files,  experts  used 
fewer  FILE  SELECT  and  ZOOM  OUT  commands  and  more  INDEX 
commands  than  novices.  Novices  apparently  failed  to  exploit 
the  additional  information  in  the  INDEX  procedure  and  as  a 
result  used  the  SCROLL  DOWN  more  often  than  experts. 
Finally,  there  was  a  large  decrease  in  use  of  the  SEARCH 
procedure  for  expert  subjects  from  Cluster  4  to  Cluster  9 
(Fisher's  LSD,  a  =  0.05).  Experts  apparently  realized  that 
the  SEARCH  procedure  was  not  the  only  ballistic  procedure  to 
locate  information  in  the  combat  support  files. 

These  are  only  a  few  of  the  comparisons  that  can  be  made 
when  analyzing  search  strategy  at  the  cluster  level.  The 
point  was  not  to  be  exhaustive  in  the  post  hoc  analysis  but 
to  determine  whether  search  strategy  analysis  was  beneficial 
at  this  level.  Indeed,  two  major  conclusions  were  drawn 
from  the  analysis  of  search  clusters.  First,  the  12  search 
tiiistero  were  useful  in  describing  the  class  and  general 
location  of  information  in  the  database.  Second,  the  12 
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search  clusters  influenced  search  performance,  and  to  a 
greater  extent,  search  strategy.  Together  these  two 
conclusions  indicate  that  component  search  tasks,  in  the 
form  of  search  clusters,  can  be  identified  through 
structured  sorting  procedures.  The  next  required  step  was 
to  extract  strategy  information  from  these  component  search 
tasks  so  that  an  online  aid  can  assist  novices  on  specific 
search  clusters.  More  specifically,  the  online  aid  should 
emphasize  meaningful  and  consistent  components  of  the  file 
selection  and  search  task  to  develop  the  information 
retrieval  skills  of  inexperienced  users  (Schneider,  1985). 

Constructing  Command- Select ion  Models 

Given  the  search  performance  and  search  strategy 
differences  between  novice  and  expert  subjects,  the 
development  of  frequency,  sequence,  and  plan-based  models  of 
command  selection  was  based  on  the  behavior  of  expert 
subjects.  As  compared  to  novices,  experts  were  a  more 
homogeneous  group  of  subjects  and  were  capable  of  generating 
a  task  structure  for  providing  intelligent  advice  about 
information  retrieval.  With  this  framework,  the  next  three 
sections  give  the  methodological  details  surrounding  the 
construction  of  the  three  command-selection  models. 


81 


Frequency  model  development.  Generating  the 
frequency-based  model  for  command  selection  closely 
paralleled  the  procedures  described  by  Elkerton  and  Williges 
(1985).  Briefly,  the  frequency-based  model  was  based  on  the 
overall  selection  of  search  procedures  by  expert  subjects. 
A  polling  procedure  similar  to  that  diagrammed  in  Table  3 
for  analyzing  search  strategies  was  used  for  determining  the 
most  frequently  used  search  procedures.  The  difference 
between  the  polling  procedure  in  Table  3  and  the  polling 
procedure  for  the  frequency-based  model  was  in  terms  of 
summing  the  polls.  Table  6  illustrates  these  changes  in  the 
polling  procedure  for  a  specific  search  problem  (Item  26)  in 
Cluster  9  (Combat  Support  Information:  Subgroups). 

As  shown  in  Table  6,  summing  the  polls  occurred  at  two 
stages.  The  first  stage  summed  the  12-vote  vectors  of  each 
subject  for  a  specific  search  problem  (Item  26).  The  result 
of  this  summation  was  an  item  profile  indicating  what  search 
procedures  were  used  most  frequently  by  expert  subjects  for 
a  search  problem.  The  second  summation  in  Table  6  shows 
that  individual  item  profiles  were  summed  to  yield  an 
overall  cluster  profile.  The  cluster  profile  indicates  the 
selection  frequency  for  specific  search  procedures  on  a  set 
of  search  problems.  For  example,  the  most  frequently 
selected  search  procedures  for  Cluster  9  were  FILE  SELECT, 
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TABLE  6 

Polling  Procedures  and  Frequency  Profiles  for  Cluster  9 
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6 

7 
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8 
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3 
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5 

4 

3 

29 

FILE  SELECT 

8 

9 
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7 

63 

followed  by  INDEX,  SEARCH,  and  SCROLL  DOWN.  The  other  11 
cluster  profiles  for  the  frequency-based  model  are  presented 
in  Appendix  K. 

The  relationship  between  the  cluster  profiles  of  experts 
and  the  type  of  information  in  the  search  problem  was 
critical  to  the  success  of  this  performance-based  approach. 
However,  inspection  of  the  example  provided  in  Table  6  (as 
well  as  other  cluster  profiles  in  Appendix  K)  suggested  a 
high  degree  of  correspondence  between  profiles  and  type  of 
cluster.  For  instance,  the  FILE  SELECT  procedure  is 
efficient  since  the  search  problems  are  contained  in  the 
combat  support  files.  Likewise,  the  INDEX  and  SEARCH 
procedures  would  be  useful  since  these  procedures  could 
quickly  access  the  subgroup  information  in  the  file.  In 
fact,  novices  could  benefit  from  a  suggestion  to  use  INDEX 
since  the  data  in  Figure  7  indicated  that  novices  may  not 
fully  utilize  this  procedure. 

How  might  this  frequency-based  information  on  the  command 
selection  of  expert  subjects  be  used?  First,  the  most 
frequently  selected  search  procedures  may  be  used  to  compare 
novice  and  expert  command  selection.  If  a  novice  chooses  an 
infrequently  selected  search  procedure,  then  an  online  aid 
could  use  the  cluster  profiles  to  base  a  decision  on  whether 
advice  or  training  should  be  given.  In  addition. 


if  the 
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online  aid  decides  to  give  advice,  the  more  frequently 
selected  search  procedures  could  be  suggested  for  use. 
Thus,  the  cluster  profiles  could  serve  both  as  a  diagnostic 
and  as  a  remedial  device  in  an  online  command- selection  aid. 

Sequence  model  development.  One  obvious  limitation  of 
the  frequency-based  model  was  the  lack  of  command- sequence 
information.  Novice  subjects  may  need  to  know  what  search 
'-'-'nmand  to  use  after  another.  To  extract  this  information, 
second-order  transition  matrices  were  constructed  to  capture 
the  sequence  of  command  use.  Table  7  shows  a  second-order 
transition  matrix  that  was  constructed  for  Cluster  9  (Combat 
Support  Information:  Subgroups).  In  Table  7  the  search 
procedure  before  the  transition  appears  as  a  row  entry  and 
the  search  procedure  after  the  transition  appears  as  a 
column  entry.  In  addition.  Table  7  contains  an  initial 
profile  that  was  used  to  describe  the  first  search  procedure 
used  by  experts.  All  of  the  entries  in  Table  7  are  total 
polls  summed  across  subjects  and  search  problems. 

The  construction  of  the  transition  matrices  and  initial 
profiles  was  based  on  polling  procedures  similar  to  those 
used  in  the  frequency-based  models.  Each  subject  on  every 
trial  was  given  a  12  X  12  matrix  of  144  votes  corresponding 
to  the  144  possible  search  command  transitions.  If  a 
subject  moved  from  one  search  procedure  to  another  during 
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TABLE  7 


Transition  Matrix  and  Initial  Profile  for  Cluster  9 


To  Initial 

Search 

From  12345  6  789101112  Profile 

1|  20  10  1  0  0  3  0  0  0  16  1  3  0 

2|  10  22  3  002  10002  0  0 

3|  122310000621  0 

4|  014222001110  1 

5|  100223004134  4 

6|  14  13  1  1  1  6  3  0  2  0  7  6  13 

7j  060003  16  00012  3 

8|  000000000000  0 

9 1  8552320020  13  5  16 

10 1  01040  12  403020  1 

11|  00203100210  23  0 

12  0  4  0  0  11  16  7  0  31  0  3  0  16 
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the  trial,  then  the  appropriate  cell  in  the  matrix  would  be 
given  a  value  of  1,  otherwise,  the  cell  would  have  a  value 
of  0.  Individual  transition  matrices  for  each  subject  were 
then  summed  to  yield  a  transition  matrix  for  a  search 
problem.  These  search  problem  matrices  were  then  summed  to 
yield  a  cluster  transition  matrix.  The  initial  profiles,  in 
contrast,  were  easier  to  construct  since  each  subject 
contributed  one  poll  to  the  initial  profile  for  each  search 
problem . 

Inspection  of  the  transition  matrix  and  initial  search 
profile  in  Table  7  provides  a  wealth  of  information  that 
could  be  valuable  to  novices.  For  example,  expert  subjects 
used  FILE  SELECT  frequently  as  the  first  search  operation. 
After  this  initial  FILE  SELECT  procedure,  expert  subjects 
frequently  selected  the  INDEX  procedure.  Thus,  when 
compared  to  the  frequency  model,  the  sequence  model  would 
provide  similar  information  in  suggesting  the  use  of  FILE 
SELECT  and  INDEX.  However,  command- sequence  information  may 
be  useful  to  novices  and  will  describe  other  situations  that 
are  not  recorded  with  the  frequency  models.  In  the 
transition  matrix  in  Table  7  for  example,  there  were  a  large 
number  of  transitions  from  ZOOM  OUT  to  FILE  SELECT.  This 
file  traversal  behavior  was  characteristic  of  Cluster  9 
(Combat  Support  Information:  Subgroups)  since  subjects  may 
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have  to  check  both  combat  support  files  to  locate  a  specific 
subgroup.  These  and  other  interesting  transitions  make  the 
sequence  model  an  attractive  representation  for  an  online 
aid.  Further  information  on  all  transition  matrices  and 
initial  profiles  for  the  12  clusters  are  available  in 
Appendix  L. 

When  implemented,  the  sequence  model  was  to  suggest  the 
most  frequently  selected  search  procedures  to  novices  based 
on  the  previous  command.  Conceptually,  once  a  previous 
search  command  was  identified,  the  row  of  the  transition 
matrix  served  as  the  frequency  profile  from  which  advice 
could  be  given.  For  instance,  if  the  search  problem  was 
part  of  Cluster  9  and  the  novice  had  previously  selected 
FILE  SELECT,  then  Row  12  in  Table  7  would  serve  as  the  basis 
for  command  selection  advice  at  that  point  in  retrieval. 

Planning  model  development.  Construction  of  the 
plan-based  model  attempted  to  simplify  the  command- sequence 
information,  but  expanded  on  the  conditions  and  situations 
for  using  a  search  procedure.  A  single  sequence  of  search 
procedures  was  needed  that  contained  both  a  strategic  and  a 
search  cluster  component.  Ideally,  an  annotated  search  plan 
was  desired  which  could  guide  a  novice  user  effectively 
through  the  database  with  a  small  number  of  search 
procedures . 
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Several  sources  of  data  were  used  to  construct  the  search 
plans  for  each  search  cluster.  Figure  8  shows  both  the  data 
and  the  process  for  search  plan  construction.  The  first 
step  in  the  process  was  to  identify  search  plans  that  were 
reported  verbally  by  experts  in  sorting  the  search  problems. 
Analysis  of  these  verbal  reports  were  by  far  the  most 
subjective  procedure  employed  in  the  search  plan  analysis 
since  the  reports  were  often  fragmentary  or  difficult  to 
understand.  However,  the  verbal  reports  did  provide  the 
experimenter  with  useful  hypotheses  about  candidate  search 
strategies  that  were  verified  with  more  objective  measures. 
For  example,  with  Cluster  9  (Combat  Support  Information: 
Subgroups)  experts  reported  using  FILE  SELECT  to  position  to 
the  combat  support  files,  then  using  INDEX,  SEARCH,  or 
SECTION  to  locate  subgroup  data. 

The  next  step  of  the  search  plan  analysis  consisted  of 
identifying  the  second-order  search  transitions  that 
corresponded  to  these  verbal  reports.  In  this  specific 
cluster,  empirical  support  was  sought  for  the  FILE  SELECT  to 
INDEX,  the  FILE  SELECT  to  SEARCH,  and  the  FILE  SELECT  to 
SECTION  transitions.  Examination  of  Table  7  shows  that 
these  transitions  were  frequently  observed  for  search 
problems  in  Cluster  9  (Combat  Support  Information: 
Subgroups).  However,  the  number  of  transitions  from  FILE 
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PLANS 


Figure  8.  Processes  for  search  plan  construction 
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SELECT  to  INDEX  was  twice  the  number  of  transitions  observed 
from  FILE  SELECT  to  SEARCH,  and  FILE  SELECT  to  SECTION.  As 
a  direct  result,  the  search  plan  analysis  focused  on  the 
FILE  SELECT  to  INDEX  search  sequence. 

The  third  step  was  to  supplement,  if  possible,  the 
second-order  transition  information  with  longer  sequences  of 
search  procedures.  For  example,  in  Cluster  9  simple  search 
paths  were  constructed  with  path  algebra  (Alty,  1984)  that 
included  the  FILE  SELECT  to  INDEX  transition.  A  simple 
search  path  is  a  sequence  of  two  search  procedures  or  more 
where  the  same  procedure  cannot  be  used  twice.  Using  these 
simple  search  paths,  frequently  used  sequences  of  search 
procedures  were  identified  that  contained  the  second-order 
transitions  under  investigation.  In  this  example,  experts 
frequently  selected  ZOOM  OUT  either  before  (45.2%)  or  after 
(32.3%)  the  transition  from  FILE  SELECT  to  INDEX.  Clearly, 
ZOOM  OUT  was  associated  closely  with  the  FILE  SELECT  to 
INDEX  transition  and  was  marked  for  further  analysis. 

The  final  step  of  the  search  plan  construction  diagrammed 
in  Figure  8  was  iterative.  The  sequence  of  search 
procedures  were  assembled  in  orders  that  could  be  used  to 
guide  a  novice  user  to  the  information  contained  in  a  search 
cluster.  For  the  information  in  Cluster  9  (Combat  Support 
Information:  Subgroups),  the  sequence:  FILE  SELECT,  INDEX, 
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and  ZOOM  OUT  was  identified  as  a  reasonable  search  plan  for 
finding  the  information.  Verbal  explanations  were  then 
written  to  supplement  the  search  plan.  The  explanations 
attempted  to  provide  information  on  how  to  use  the  specific 
search  procedures  to  retrieve  data  contained  in  the  search 
cluster.  The  explanation  could  include  information  on  the 
general  location,  of  the  search  problem  in  the  databr.se,  the 
keywords  to  look  for  in  files,  and  other  procedural 
directions  such  as  when  to  repeat  steps  of  search  plan. 
Information  for  the  explanation  was  drawn  from  the  verbal 
reports,  the  search  cluster  descriptions,  and  the  actual 
search  problems.  Next,  the  search  plans  were  refined 
iteratively  in  an  attempt  to  provide  good  advice  for  all 
problems  in  a  search  cluster.  Changes  that  could  occur  to 
the  search  plans  during  iteration  included  changes  in  the 
wording,  and  adding  a  search  procedure  to  improve  the 
coverage  for  the  set  of  search  problems. 

The  final  search  plan  for  Cluster  9  (Combat  Support 
Information:  Subgroups)  is  displayed  in  Table  8.  In 
addition,  the  other  search  plans  generated  for  the  other  11 
search  clusters  are  available  in  Appendix  M.  As  can  be  seen 
in  Table  8,  the  search  plan  contains  both  a  sequence  of 
search  procedures  and  a  explanation  for  using  the 
procedures.  The  decision  was  made  to  include  ZOOM  OUT  as  a 


TABLE  8 


Final  Search  Plan  for  Cluster  9 

Steps  Description 

1.  FILE  SELECT 

(to  a  combat  support  file) 

2 .  INDEX 

(to  locate  subgroup  information) 


3. 


If  target  is  not  located  ZOOM  OUT 
(and  repeat  steps  1  and  2) 
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conditional  step  since  novices  might  not  know  which  combat 
support  file  to  look  in.  Comparing  this  planning  model  with 
the  previous  performance  models  (frequency  and  sequence) 
illustrates  the  additional  information  that  is  available  on 
general  location  of  the  search  problems,  and  the  general 
type  of  information  to  be  searched  for.  In  essence,  the 
goal  of  the  planning  model  was  to  make  explicit  the 
component  search  tasks  that  were  identified  by  experts. 

Summary 

Four  major  results  were  obtained  in  Experiment  1.  The 
first  two  results  focused  on  the  differences  between  expert 
and  novice  users.  Of  primary  importance  was  the 
demonstration  of  overall  differences  in  the  search 
performance  of  expert  and  novice  users.  Demonstrating  this 
difference  was  required  before  an  online  aid  could 
implemented  and  evaluated.  The  second  and  somewhat 
surprising  result  was  the  heterogeneity  of  novice  search 
performance  that  contributed  to  the  performance  differences 
and  obscured  the  strategy  differences  between  novices  and 
experts.  The  implication  was  that  novices  could  be  viewed 
in  future  evaluations  as  two  groups  of  subjects:  fast  and 
slow  novices.  The  theoretical  justification  rests  on  data 
surrounding  the  acquisition  of  high-performance  skills 
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(Schneider,  1985).  This  research  predicts  large  differences 
m  the  skills  of  new  users  of  complex  systems. 

The  last  two  results  centered  on  the  development  of 
methods  to  alleviate  these  differences  through  online 
command-selection  aids.  Specifically,  a  component  task 
model  was  developed  using  the  sorting  and  clustering 
procedures  that  identified  12  sets  of  homogeneous  search 
problems.  These  12  search  clusters  were  found  to  be  useful 
in  describing  the  more  cognitive  aspects  of  information 
retrieval.  Most  importantly  in  this  experiment,  three 
command-selection  models  were  described  and  constructed  that 
could  be  used  for  providing  novice  users  different  types  of 
assistance  or  instruction  in  the  information  retrieval  task. 


EXPERIMENT  2:  EVALUATION  OF  THE  COMMAND- 
SELECTION  AIDS 


Approach 

The  differences  between  novices  and  experts  in  the 
information  retrieval  task  suggests  that  providing  novice 
users  an  online  aid  may  be  beneficial.  As  was  stated  in  the 
previous  experiment,  two  potential  benefits  could  result 
from  effective  aiding.  First.  slower  novices  that  are 
provided  an  online  aid  could  improve  their  information 
retrieval  skills.  Second,  aiding  could  result  in  a  more 
homogeneous  group  of  novice  users  that  would  be  easier  to 
support.  These  are  extremely  important  goals  since  the 
differences  between  slow  novices  and  other  subjects  were 
large  and  persisted  over  several  days.  Slow  novices  were 
not  capable  of  closing  the  "skill  gap"  between  themselves 
and  other  more  proficient  users. 

Unfortunately,  novice  heterogeneity  could  pose  problems 
for  an  online  aid.  Novices  in  this  task  have  a  wide  variety 
of  needs  that  have  to  be  satisfied.  Some  novices  may 
require  a  large  amount  of  detail  in  selecting  commands, 
while  others  might  only  need  hints.  The  three  different 
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command-selection  models  were  constructed  to  answer  this 
question.  Progressing  from  the  frequency  to  the  sequence  to 
the  plan-based  model  adds  more  information  to  the 
command-selection  aid.  In  this  experiment  the  question  was 
asked:  What  type  of  command-selection  information  is  needed 
by  the  novice  user? 

Similarly,  novice  heterogeneity  may  pose  problems  in 
defining  a  human-computer  dialogue  for  an  online  aid.  Slow 
and  inefficient  novice  users  may  be  unaware  of  their 
computing  shortcomings.  These  types  of  users  may  require 
computer-initiated  advice  to  call  attention  to  their 
problems.  Fast  and  efficient  novice  users,  however,  may  not 
reguire  and  may  even  be  slowed  by  computer-initiated  advice. 
These  fast  novices  probably  have  an  accurate  understanding 
of  their  skills  and  will  benefit  from  advice  on  only  rare 
occasions.  Thus,  a  dialogue  that  is  more  user  controlled 
(user-  or  mixed-initiated)  may  be  more  helpful  and  less 
intrusive  to  these  faster  novices. 

To  evaluate  these  issues  an  experiment  was  conducted 
using  novice  subjects  that  received  various  combinations  of 
command-selection  aids  (frequency,  sequence,  and  plan-based 
models)  and  dialogue  initiatives  (computer,  user,  and 
mixed) .  The  approach  was  to  determine  the  effectiveness  of 
the  three  command- selection  aids  with  the  three  dialogue 
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initiatives.  Effectiveness  was  to  be  assessed  through 
comparison  of  aided  novices  with  the  novice  and  expert 
groups  of  Experiment  1.  Both  aided  and  unaided  search 
performance  and  search  strategy  were  to  be  evaluated  in  this 
experiment.  The  goal  of  this  approach  was  to  determine 
which  online  aids  were  effective  in  assisting  users  in  the 
information  retrieval  task,  as  well  as  which  online  aids 
resulted  in  the  most  learning.  Finally,  all  novice  subjects 
in  this  experiment  were  asked  to  evaluate  the 
command-selection  aids  subjectively.  In  online  advisory 
systems,  user  satisfaction  is  of  major  importance  since  the 
perceived  quality  of  the  advice  could  affect  user 
interaction  with  the  aid. 

Method 

Subi ects .  An  additional  43  novices  volunteered  for  the 
experiment.  The  procedures  and  criteria  for  the  subjects 
were  the  same  as  Experiment  1.  Only  one  novice  subject  in 
this  experiment  was  excluded  for  failure  to  find  the 
training  information  fast  enough  (under  300  seconds, 
average).  Another  6  novice  subjects  were  used  as  pretest 
subjects  resulting  in  a  sample  size  of  36  novices  for  the 
experiment . 


98 


Experimental  design.  Table  9  shows  the  design  of 
Experiment  2.  The  factorial  experiment  was  a  3X3 
mixed-factor  design  with  command- selection  model  as  the 
~_tween-subjects  variable  and  dialogue  initiative  as  the 
within-subjects  variable.  Table  9  also  shows  that  the 
novices  and  experts  from  Experiment  1  served  as  control 
groups  for  the  command- selection  aids  evaluated  in 
Experiment  2.  Therefore,  to  facilitate  comparisons  with  the 
control  novices  and  experts,  all  procedures  and  materials  in 
Experiment  2  were  the  same  as  Experiment  1  with  exception  of 
the  online  aiding  manipulations. 

Each  of  the  36  novice  subjects  in  Experiment  2  was 
assigned  randomly  to  one  of  three  command-selection  aids 
shown  in  Table  9.  After  receiving  two  initial  training 
sessions  on  the  information  retrieval  task,  all  aided 
novices  could  receive  command-selection  advice  through  the 
three  different  dialogue  initiatives  on  three  separate 
sessions.  Before  the  start  of  each  experimental  session, 
all  36  novices  received  instructions  both  on  the  type  of 
information  presented  by  the  online  aid  (Appendix  N)  and  the 
procedures  for  accessing  the  command-selection  advice 
(Appendix  0) .  In  these  instructions  subjects  were 
encouraged  to  use  the  online  aid,  but  were  told  that  speed 
of  information  retrieval  was  still  important. 
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TABLE  9 

Experimental  Design 

Aided  Novices  (Experiment  2) 


Dialogue  Initiative 


Command- Select ion 
Model 

User 

Computer 

Mixed 

Frequency 

Subjects 

25  - 

36 

Sequence 

Subjects 

37  - 

48 

Plans 

Subjects 

49  - 

60 

Control  Groups  (Experiment  1) 


Novices 

Subjects 

1-12 

Experts 

Subjects 

13  -  24 
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There  were  two  reasons  for  selecting  dialogue-initiative 
as  a  within-subj ects  variable  given  the  potential  for 
learning  and  transfer  effects.  First,  the  experiment  was 
infeasible  to  conduct  with  dialogue  initiative  as  a 
between-subjects  variable  due  to  the  length  of  data 
collection  (5-6  days).  Second,  changing  the  dialogue 
initiative  has  been  suggested  as  a  method  for  adapting  the 
interface  to  the  user  (Maskery,  1984).  For  example,  a 
computer-ini tiated  dialogue  could  change  to  a  user-initiated 
or  a  mixed-initiated  dialogue  as  the  user  becomes  more 
proficient  with  the  interface.  To  further  study  these 
sequence  effects,  the  presentation  orders  of  the  dialogue 
initiatives  were  completely  counterbalanced  and  are  shown  in 
Table  10.  Two  novices  from  each  command- selection  group 
were  assigned  randomly  to  these  presentation  orders. 

Table  10  also  illustrates  that  aided  novices  received  a 
final  unaided  transfer  session  after  completing  the  three 
dialogues.  The  goal  of  this  final  session  was  to  assess  the 
training  effectiveness  of  the  three  command- selection  aids. 
Therefore,  in  addition  to  evaluating  tne  ability  of  the 
command-selection  aids  to  assist  novices  on  the  current 
retrieval  task,  the  experiment  was  also  a  transfer  of 
training  design. 
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TABLE  10 

Presentation  Orders  of  the  Dialogue  Initiatives 


Session 


Presentation 

Order 

3 

4 

5 

6 

CMU 

Computer 

Mixed 

User 

Unaided 

CUM 

Computer 

User 

Mixed 

Unaided 

MCU 

Mixed 

Computer 

User 

Unaided 

MUC 

Mixed 

User 

Computer 

Unaided 

UCM 

User 

Computer 

Mixed 

Unaided 

UMC 

User 

Mixed 

Computer 

Unaided 
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Implementing  the  command- selection  models .  Implementing 
the  frequency,  sequence,  and  plan-based  models  to  provide 
advice  to  novice  users  required  that  the  search  cluster  be 
identified,  the  command-selection  information  in  Appendices 
K,  L,  and  i>4  be  presented  visually,  and  decisions  be  made 
about  how  novices  should  be  monitored  and  when  advice  should 
be  given.  The  first  problem  was  trivial,  while  the  later 
two  problems  were  significant  issues  that  were  addressed 
when  implementing  each  command-selection  model. 

Decisions  on  how  the  frequency  command- selection  model 
was  implemented  were  based  largely  on  the  results  of 
Elkerton  (1984).  The  results  of  this  study  indicated  that  a 
frequency  profile  of  command  use  may  be  helpful  when 
implemented  as  a  ranked  set  of  suggested  search  procedures 
on  the  touch  keypad.  The  results  also  indicated  that  the 
number  of  search  procedures  used  to  determine  when  the 
novice  needed  advice  was  critical  to  the  success  the  online 
aid.  Specifically,  Elkerton  (1984)  found  that  a  smaller 
number  of  frequently  selected  search  procedures  (2,  3,  and 
4)  should  be  used  to  determine  when  novices  needed  advice 
since  a  larger  number  (5)  would  result  in  advice  being 
presented  late  in  the  search  trial.  In  other  words,  a 
strict  criterion  for  the  search  procedures  that  can  be 
selected  by  novices  should  be  used  for  frequency-based 
advice  to  be  presented  at  the  start  of  the  search  trial. 
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Although  the  results  of  Elkerton  (1984)  were  difficult  to 
generalize  to  the  more  complex  file  selection  and  search 
task,  the  method  of  providing  a  ranked  set  of  highlighted 
search  procedures  on  the  touch  keypad  was  followed. 
Furthermore,  the  suggested  strict  criterion  for  comparing 
the  selection  of  search  procedures  by  novices  and  experts 
was  heeded  for  conditions  where  the  computer  would  provide 
frequency-based  advice.  An  initial  setting  of  3  was 
pretested  and  adopted  as  the  number  of  frequently  selected 
search  procedures  that  were  accepted  and  presented  by  the 
online  aid  for  a  specific  search  cluster.  This  criterion 
allowed  novices  to  traverse  the  file  hierarchy  and  to  use 
ballistic  search  procedures  without  receiving  these  search 
procedures  as  advice.  Overall  selection  frequencies  were 
used  to  break  ties  when  extracting  the  frequently  selected 
'--arch  procedures  from  the  cluster  profiles  in  Appendix  K. 

An  example  of  the  frequency-based  advice  for  the  search 
problems  in  Cluster  9  is  presented  in  Figure  9.  In  addition 
to  ranking  search  procedures  that  were  selected  frequently 
by  experts,  the  online  aid  also  highlighted  these  search 
procedures  on  the  touch  keypad.  Novices  could  use  this 
advice  or  choose  any  other  search  procedure.  Advice  for  the 
frequency  model  could  be  presented  only  once  and  would 
remain  on  the  primary  display  until  completion  of  the  trial. 
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Figure  9.  Frequency-based  advice  presented  to  novice 
subjects  for  search  problems  in  Cluster  9. 
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The  implementation  of  the  sequence  model  for 
w^nunand-selection  advice  closely  paralleled  the  frequency 
model  in  the  number  of  presented  and  accepted  search 
procedures.  As  an  example.  Figure  10  shows  the 
command- sequence  advice  given  to  a  novice  after  using  FILE 
SELECT  as  the  initial  search  procedure  to  locate  information 
in  Cluster  9  (Combat  Support  Information:  Subgroups). 
Instead  of  offering  search  procedures  that  were  used  most 
frequently  overall,  the  sequence  model  suggested  search 
procedures  that  were  selected  frequently  after  the  use  of 
FILE  SELECT  (i.e.,  INDEX,  SEARCH,  and  SECTION).  These 
suggested  search  procedures  were  extracted  from  the  last  row 
of  the  transition  matrix  shown  in  Table  7  and  represent  the 
search  procedures  that  experts  used  most  frequently  after 
the  selection  of  FILE  SELECT.  Similar  to  the  frequency 
model,  ties  were  broken  in  the  12  X  12  matrices  and  initial 
search  profiles  (Appendix  L)  with  an  overall  transition 
matrix  and  initial  profile  based  on  all  clusters.  Novices 
could  choose  whether  to  use  the  advice  or  not.  However, 
unlike  the  frequency  advice,  command- sequence  advice  could 
be  presented  several  times  during  a  trial.  For  example, 
novices  could  receive  advice  at  the  start  of  the  trial  based 
on  the  initial  search  profile  and  then  could  receive  further 
command- selection  advice  based  on  the  12  X  12  transition 


matrix. 
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Figure  10.  Command- sequence  advice  presented  to  novice 
subjects  for  search  problems  in  Cluster  9. 
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The  implementation  of  the  plan-based  model  is  shown  in 
Figure  11.  As  can  be  seen  in  Figure  11,  the  search  plan 
developed  for  Cluster  9  was  presented  in  the  work  area  of 
the  primary  display.  Contained  in  this  plan  was  a  sequence 
of  search  procedures  (steps)  with  appropriate  verbal 
explanations.  The  search  procedures  in  the  plan  were 
capitalized  and  highlighted.  Moreover,  the  appropriate 
search  procedures  on  the  touch  keypad  were  highlighted  and 
numbered  according  to  the  step  in  the  plan.  All  plans  could 
be  presented  only  once  and  would  remain  on  the  primary 
display  until  the  end  of  the  trial.  If  the  work  area  was 
required  by  another  search  procedure  (e.g.,  FILE  SELECT), 
the  planning  steps  were  erased  and  automatically  displayed 
again  after  completion  of  the  search  procedure. 

Implementing  the  dialogue  initiatives.  Closely 
associated  with  the  command-selection  models  were  the 
implementations  of  the  user-,  computer-,  and 
mixed-initiative  dialogues.  Each  of  these  dialogues 
provided  a  different  method  for  accessing  the 
command-selection  advice. 

The  design  of  user-initiated  advice  was  straightforward 
with  the  frequency  and  plan-based  models.  Novices  were 
allowed  in  these  conditions  to  press  a  HELP  function  key  on 
the  VT100  keyboard  at  any  time  during  the  search  trial  to 
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Figure  11.  Plan-based  advice  presented  to  novice  subjects 
for  search  problems  in  Cluster  9 . 
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receive  either  the  frequency  or  the  plan-based  advice.  For 
the  sequence  model  the  question  was  asked  whether  novices 
should  be  allowed  to  press  HELP  more  than  once  a  trial.  The 
uecision  was  made  to  implement  this  capability  since  a 
possible  strength  of  the  sequence  model  was  to  provide 
command-selection  advice  at  any  point  during  search. 

The  design  of  the  computer-initiated  advice  was  more 
complicated  for  all  command-selection  aids.  Indeed,  the 
topics  of  how  novices  should  be  monitored  and  when 
command- selection  advice  should  be  given  were  covered 
briefly  with  the  implementation  of  frequency-based  models. 
To  expand  on  this  discussion,  the  cluster  profiles  of 
experts  (Appendix  K)  served  as  a  differential  model  (Burton 
and  Brown,  1982)  for  comparing  expert  and  novice  search 
procedure  selection.  Command-selection  advice  was  given  if 
a  novice  subject  used  any  search  procedure  other  than  the 
top  three  search  procedures  selected  by  experts.  This 
procedure  was  very  similar  to  the  computer-initiated 
techniques  employed  by  Elkerton  (1984).  However,  instead  of 
the  individual  search  problem  profiles  used  by  Elkerton 
(1984),  this  research  used  the  more  general  cluster 
profiles. 

Providing  computer-initiated  advice  with  the  sequence  and 
plan-based  models  required  that  the  selection  of  search 
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procedures  be  monitored  in  terms  of  command- sequence  rather 
than  frequency.  To  accomplish  this  goal,  the 
command-selection  aids  used  the  12  X  12  transition  matrix 
and  the  initial  search  profile  for  a  specific  search  cluster 
(Appendix  L)  as  the  differential  model  to  compare  expert  and 
novice  search  procedure  selection.  Similar  to  the  frequency 
model,  a  strict  criterion  of  the  three  most  frequently 
selected  search  procedures  was  used  to  determine  when  to 
give  command-selection  advice. 

For  example,  if  the  novice  was  searching  for  information 
represented  by  the  initial  search  profile  in  Table  7  (Combat 
Support  Information:  Subgroups),  then  the  three  acceptable 
search  procedures  would  be  FILE  SELECT,  INDEX,  and  SEARCH. 
Choosing  any  other  search  procedure  initially  would  lead  to 
computer-initiated  presentation  of  either  the  sequence  or 
plan-based  advice.  If  plan-based  information  was  presented, 
then  additional  command- selection  advice  could  not  be 
given.  However,  if  the  sequence  model  was  used,  then 
additional  command- sequence  advice  could  be  presented  to  the 
novice . 

With  the  sequence  model,  if  the  novice  took  the 
highlighted  advice  and  used  FILE  SELECT  to  position  to 
another  file,  then  the  three  most  frequently  selected  search 
procedures  in  row  12  of  the  transition  matrix  in  Table  7 
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would  be  used  to  judge  the  selection  of  the  next  search 
procedure.  As  seen  in  row  12  of  the  transition  matrix,  the 
novice  would  receive  command- selection  advice  only  when 
using  search  procedures  other  than  INDEX,  SEARCH,  and 
SECTION.  This  technique  of  comparing  the  search  procedure 
selection  of  novice  users  based  on  the  previously  selected 
command  would  continue  until  the  end  of  the  trial. 

In  pretesting,  however,  computer-initiated  advice  with 
the  sequence  model  was  disastrous.  The  command-selection 
advice  was  presented  too  frequently  since  the  sequence  model 
could  provide  advice  at  every  step  in  the  search  trial. 
Novices  would  become  overwhelmed  with  the  amount  of  advice. 
To  prevent  this  information  overload,  computer-initiated 
advice  with  the  sequence  model  was  constrained  to  provide 
advice  only  when  the  novice  entered  a  new  file  in  the 
hierarchy. 

The  final  dialogue  was  a  mixture  of  computer-  and 
user-initiated  dialogues  that  attempted  to  maintain  user 
control.  In  this  mixed-initiative  dialogue,  novices  could 
request  command- selection  advice  by  pressing  the  HELP 
function  key  on  the  VT100  keyboard.  In  addition,  the 
command-selection  aid  would  monitor  the  use  of  search 
procedures  with  the  computer-initiated  techniques  just 
described  and  would  suggest  to  the  novice  that 
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command-selection  advice  was  available.  Suggesting  the 
availability  of  advice  was  implemented  by  presenting  the 
highlighted  message  "If  you  want  advice  press  HELP"  directly 
below  the  touch  keypad  on  the  primary  display.  The  message 
was  designed  be  informative,  but  not  intrusive  to  the  novice 
user.  As  a  result,  the  novice  user  maintained  control  of 
the  dialogue  with  the  online  aid  and  could  request  the 
display  of  command-selection  advice  at  later  and  possibly 
more  appropriate  times. 

Subjective  rating  scales .  In  addition  to  the  embedded 
performance  measurement,  subjective  satisfaction  with  the 
online  aids  was  assessed  at  the  end  of  each  information 
retrieval  session.  The  instrument  for  user  evaluation  of 
the  online  aid  consisted  of  a  set  of  12  bipolar  rating 
scales  and  a  final  ranking  of  the  three  dialogue 
initiatives.  The  12  bipolar  scales  consisted  of  an  overall 
satisfaction  scale  and  11  individual  adjective  scales  for 
determining  specific  subjective  characteristics  of  the 
command- selection  aids. 

The  12  pairs  of  bipolar  adjectives  and  the  anchors  for 
the  accompanying  seven-point  scale  are  presented  in  Table 
11.  The  first  8  individual  bipolar  adjectives  (2-9)  were 
selected  from  a  study  toy  Coleman  (1985)  that  developed 
bipolar  adjective  scales  for  the  evaluation  of  text  editors 


TABLE  11 


Bipolar  Adjective  Scales  for  User  Evaluation  of  the  Online 
Aids 


Overall  Satisfaction 


1.  UNSATISFACTORY 


SATISFACTORY 


Individual  Bipolar  Adjectives 


2  .  UNNATURAL . NATURAL 

3  .  UNDEPENDABLE . DEPENDABLE 

4.  COMPLEX . SIMPLE 

5.  SLOW . FAST 

6  .  I NCOMP LETE . COMPLETE 

7.  DISGUSTING . PLEASING 

8.  UNCOOPERATIVE . COOPERATIVE 

9.  USELESS . USEFUL 

10.  UNADAPTIVE . ADAPTIVE 

11.  STUPID . INTELLIGENT 

12.  HINDERING . HELPFUL 


Anchors  for  the  Seven-Point  Scale 


extremely  quite  slightly  neutral  slightly  quite  extremely 
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by  novice  users.  These  8  scales  were  found  to  be  sensitive 
indicators  of  novice  satisfaction  with  text  editors.  An 
additional  3  bipolar  adjectives  (10  -  12)  were  added  to  the 
evaluation  instrument  since  the  application  task 
(information  retrieval)  and  the  evaluated  software  (an 
online  advisory  system)  differed  from  the  text-editing 
interfaces  evaluated  by  Coleman  (1985).  In  contrast  to  the 
8  individual  scales  selected  from  empirical  research,  the  3 
additional  bipolar  adjectives  (UNADAPTIVE-ADAPTIVE, 
STUPID- INTELLIGENT,  and  HINDERING-HELPFUL)  were  chosen 
intuitively  to  assess  the  advice-giving  characteristics  of 
an  online  aid. 

All  12  bipolar  adjective  scales  were  administered  at  the 
end  of  each  aided  session  on  a  VT100  display  with  a  Carroll 
Technology  touch-entry  device.  An  example  of  a  displayed 
bipolar  scale  is  shown  in  Figure  12.  As  illustrated,  the 
"-’jective  pairs  were  shown  individually  a  display  at  a  time. 
The  left-right  positioning  of  the  adjective  and  its  antonym 
was  randomized  with  the  restriction  that  an  equal  number  of 
"positive"  adjectives  (e.g.,  FAST)  occurred  on  both  the  left 
and  right  sides  of  the  scale.  Furthermore,  the  repeated 
administrations  of  the  evaluation  instrument  for  each 
dialogue  initiative  required  randomizing  the  presentation 
order  of  the  11  individual  bipolar  scales. 
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FILE  SEARCH  ASSISTANT 

Please  evaluate  the  file  search  assistant  in  tents 
of  overall  satisfaction. 

Use  the  arrou  keys  to  move  the  cursor  and  hit  the 
RETURN  key  on  the  wain  keyboard  to  enter  your  rating. 

UNSATISFACTORY  SATISFACTORY 


I — i — * — t — i — i — i 

extremely  quite  slightly  neutral  slightly  quite  extremely 


Figure  12.  Display  of  the  bipolar  adjective  rating  scale 
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Subjects  were  instructed  to  evaluate  the  file  search 
assistance  by  moving  a  cursor  with  the  displayed  arrow  keys 
to  one  of  the  seven  positions  corresponding  to  the  seven 
verbal  anchors.  Cursor  position  was  set  initially  at 
"neutral".  All  subjects  were  instructed  and  given  practice 
on  the  software  for  evaluating  the  online  aids.  Both  the 
instructions  and  examples  for  practice  are  given  in  Appendix 
P. 

The  final  subjective  evaluation  technique  asked  subjects 
to  rank  order  the  dialogue  initiatives  in  terms  of 
preference  from  1  to  3 .  A  1  indicated  the  most  preferred 
dialogue  and  a  3  indicated  the  least  preferred  dialogue. 
This  final  evaluation  occurred  at  the  end  of  the  final  day 
of  the  experiment  (Session  6).  The  form  for  collecting  the 
ranked  dialogue  preferences  is  contained  in  Appendix  Q. 

Results  and  Discussion 

Novice  subject  comparability.  Comparisons  between  the 
control  novices  and  the  aided  novices  were  necessary  to 
evaluate  online  aiding.  As  a  result,  questions  were  raised 
regarding  the  similarity  of  novice  subject  groups.  In 
particular,  the  heterogeneity  of  novice  search  performance 
in  Experiment  1  may  have  resulted  from  a  sampling  problem 
whereby  two  distinct  groups  of  novices  were  selected.  The 


117 


question  posed  in  this  analysis  was  whether  the  sample  of  36 
novices  in  Experiment  2  exhibited  similar  information 
retrieval  behavior. 

To  test  this  sampling  hypothesis,  the  last  set  of  12 
training  trials  was  analyzed  for  differences  in  average 
search  performance  and  the  variability  in  search  performance 
of  the  4  novice  subject  groups  (control  novices,  frequency 
novices,  sequence  novices,  and  plan  novices).  The  results 
of  th  one-way  ANOVAs  on  total  operations,  number  of 
different  operations,  total  time,  and  total  movement 
revealed  no  significant  mean  differences  (p  >  0.05)  between 
novice  subject  groups.  In  addition,  the  results  of  the 
Moses  rank-like  test  (Hollander  and  Wolfe,  1973)  for 
detecting  differences  in  search  variability  found  no 
significant  differences  (p  >  0.05)  for  the  four  dependent 
measures  when  comparing  the  control  novice  group  to  each  of 
the  aided  groups.  In  total,  there  appeared  to  be  no  support 
for  the  idea  that  poor  overall  performance  of  the  control 
novices  was  due  to  a  sampling  problem.  Based  on  the 
training  data,  novices  in  the  second  experiment  were 
comparable  to  control  novices  in  terms  of  speed  and 
variability  in  information  retrieval. 

Aided  search  performance :  Session  and  model  effects. 
Aided  search  performance  was  analyzed  first  to  determine 
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whether  there  were  any  significant  reductions  in  total 
operations,  different  operations,  total  time,  or  total 
movement  over  Sessions  3  through  5  as  a  result  of  the 
command-selection  aids.  In  this  analysis,  novices  receiving 
the  three  command-selection  models  (frequency,  sequence,  and 
plan-based)  were  compared  against  the  control  novices  and 
experts  of  Experiment  1.  Sessions  were  analyzed  to 
determine  whether  there  were  any  differential  learning 
effects  with  the  online  command-selection  aids. 

The  search  performance  measures  which  exhibited  a 
significant  difference  between  the  five  subject  groups 
(frequency  novices,  sequence  novices,  plan-based  novices, 
control  novices,  and  experts)  were  total  operations 
( F[ 4, 55 ]  =  2.76,  p  =  0.0367)  and  total  time  (F[4,55]  =  4.50, 
p  =  0.0032).  The  subject  groups  means  and  the  results  of 
the  Newman-Keuls  analyses  (a  =  0.05)  for  total  operations 
and  total  time  are  presented  in  Table  12.  The  means 
indicated  no  overall  differences  between  the  novices  for 
both  total  operations  and  time,  and  a  significant  difference 
between  novices  and  experts  for  total  time.  The  lack  of  any 
mean  differences  between  subject  groups  for  total  operations 
was  probably  a  result  of  the  conservative  control  of  alpha 
error  by  the  Newman-Keuls  procedure  and  the  large 
variability  of  novice  search  performance. 
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TABLE  12 


Subject  Group  Means  for  Overall  Aided  Search  Performance 


Dependent  Sequence  Control  Frequency  P lan 

Measure  Novices  Novices  Novices  Novices  Experts 

Total  103.07  95.95  94.32  93.21  67.67 

Time  ___ - - —  - 


Total  10.9  11.0  9.9  9.1 

Operations _ _ _ _ 


Means  with  a  common  horizontal  are  not  significantly 
different,  Newman-Keuls  (a  =  0.05). 
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The  large  session  effects  for  total  operations 
(£[2,110]  =  35.38,  E  =  0.0001),  different  operations 
( F [ 2 , 110 ]  =  23.12,  E  =  0.0001),  total  time 
( F [ 2 , 110 ]  =72.79,  e  =  0.0001),  and  total  movement 
(F[2,110]  =  23.96,  e  -  0.0001)  paired  with  the  lack  of  any 
session  by  subject  group  interactions  (e  >  0.05)  indicates 
that  a  large  amount  of  learning  occurs  irrespective  of  the 
type  of  aiding  received.  Thus,  the  session  effects  shown  in 
Experiment  1  would  be  representative  of  the  learning  effects 
in  this  experiment  with  the  subject  group  means  adjusted 
accordingly. 

The  overall  analyses  of  aided  search  performance 
indicated  that  aided  novices  performed  no  differently  from 
control  novices.  From  this  perspective,  the 
command-selection  aids  failed  to  assist  novice  users. 
However,  the  heterogeneity  of  novice  search  performance  must 
be  considered.  Fast  and  slow  novices  should  be  compared 
against  aided  novices.  Conducting  these  unbalanced  ANOVAs 
would  indicate  whether  the  command-selection  aids  were 
capable  of  assisting  slower  novices  without  interfering  with 
the  fast  novices.  In  performing  these  analyses,  the 
assumption  was  made  that  both  fast  and  slow  novices  were 
included  in  the  aided  groups. 
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Consistent  with  the  previous  balanced  ANOVAs  on  sessions 
and  subject  groups,  the  results  of  these  unbalanced  ANOVAs 
revealed  large  session  effects  (p  =  0.0001)  for  all  search 
measures.  The  effect  of  subject  group  was  more  pronounced 
as  well,  with  total  operations  (F[5,54]  =  5.93,  p  =  0.0002), 
different  operations  (F[5,54]  =  3.29,  p  =  0.0116),  total 
time  (F[5,54]  =  8.14,  p  =  0.0001),  and  total  movement 
(F[5,54]  =  3.62,  p  =  0.0069)  yielding  significant 
differences  between  search  means.  Finally,  the  unbalanced 
M’OVAs  revealed  no  significant  (p  <  0.05)  interactions  of 
subject  groups  and  sessions. 

Figure  13  shows  the  differences  between  subject  groups 
and  the  results  of  the  Newman-Keuls  (a  =  0.05)  analyses  on 
the  means  for  total  operations  and  total  time.  The  means 
and  post  hoc  analyses  for  different  operations  and  total 
movement  were  not  displayed  in  Figure  13  since  the  pattern 
of  results  were  similar  to  total  operations.  The  means  and 
Newman-Keuls  tests  shown  in  Figure  13  convey  an 
interpretation  of  the  aiding  data  that  differs  from  the 
overall  analysis  reported  in  Table  13.  In  general,  online 
aiding  improved  the  performance  of  slower  novices  without 
interfering  with  the  faster  novices.  These  results  were 
clear  for  total  operations,  different  operations,  and  total 
movement,  but  were  only  partially  true  for  total  time. 
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SLOW  SEQUENCE  FREQUENCY  PLAN  FAST  EXPERTS 

NOVICES  NOVICES  NOVICES  NOVICES  NOVICES 


SLOW  SEQUENCE  FREQUENCY 

NOVICES  NOVICES  NOVICES 


FAST  EXPERTS 
NOVICES 


Figure  13.  Comparison  of  fast  novices,  slow  novices,  aided 
novices  and  experts  for  total  operations  and  time.  Groups 
Wlth  a  common  horizontal  line  are  not  significantly 
different.  Newman-Keuls  (a  =  0.05). 
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As  shown  in  Figure  13,  novices  receiving  the  frequency 
and  plan-based  advice  were  performing  at  the  level  of  fast 
novices  in  terms  of  total  time.  Sequence  novices,  although 
finding  information  as  fast  as  other  aided  novices,  appeared 
to  be  having  difficulty  and  were  retrieving  information  only 
slightly  faster  than  slow  novices.  These  results  imply  that 
frequency  and  plan-based  models  could  be  useful  for  aiding 
the  slower  novices  without  interfering  significantly  with 
the  faster  novices.  Sequence  models,  however,  were  not  as 
helpful  in  assisting  the  novice  user  with  information 
retrieval  and  may  even  be  intrusive  for  faster  novices. 

The  slight  superiority  of  the  frequency  and  plan-based 
models  probably  rests  on  simplicity.  Frequency  profiles  and 
plans,  as  implemented  in  this  experiment,  were  presented 
only  once  a  trial  and  remained  on  the  display  until 
completion  of  the  trial.  The  permanence  of  the  advice 
allowed  subjects  time  to  use  and  integrate  the 
command-selection  information  with  their  own  search 
strategies.  The  sequence-based  advice,  in  contrast,  was 
more  dynamic  and  may  not  have  allowed  this  integration. 
With  sequence-based  advice,  novices  were  provided  what 
search  procedure  to  use  next  based  on  the  previously 
selected  search  procedure.  Thus,  a  large  amount  of 
constantly  changing  information  was  presented  to  the  novice. 
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making  it  extremely  difficult  to  understand  the  underlying 
structure  of  the  command-selection  advice. 

Aided  search  performance :  Pi  alocrue  and  order  effects . 
Frequency  and  plan-based  models  seem  to  be  capable  of 
assisting  some  novice  subjects.  Therefore,  the  next  step 
was  to  determine  how  the  advice  should  be  delivered  to  the 
novice  users  for  effective  assistance.  The  dialogue 
initiatives  (user,  computer,  and  mixed),  the  presentation 
order  of  the  dialogues,  and  whether  these  two  factors 
interacted  with  command-selection  model  were  of  interest. 
The  initial  three-way  ANOVAs  focused  entirely  upon  aided 
novices . 

The  results  of  the  4  ANOVAs  on  search  performance 
revealed  significant  effects  (p  <  0.05)  for  both  dialogue 
initiative  and  the  interaction  of  dialogue  initiative  by 
presentation  order.  Consistent  with  the  previous  analyses, 
type  of  command-selection  model  did  not  lead  to  any 
significant  differences  in  search  performance  between  the 
three  aided  groups.  Moreover,  interactions  with  the  type  of 
command-selection  model  were  nonexistent  with  the  two 
dialogue  variables  (initiative  and  presentation  order). 

Dialogue  initiative  was  significant  for  total  operations 
(£[2,36]  =  7.26,  p  =  0.0022),  different  operations 


(£[2,36]  =  10.28,  p  =  0.0003),  total  time  (£[2,36]  =  5.03 
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g  =  0.0119),  and  total  movement  (F[2,36]  =  4.48, 
g  =  0.0183).  The  means  and  the  results  of  the  Newman-Keuls 
procedure  (a  =  0.05)  for  the  total  time  and  different 
operations  are  presented  in  Figure  14.  The  results  for 
total  operations  and  total  movement  were  similar  to  total 
time  and  are  not  presented. 

Figure  14  illustrates  that  mixed-initiative  dialogues  led 
to  the  worst  performance  with  the  command-selection  aids. 
For  every  dependent  measure,  user-  and  computer-initiated 
dialogues  were  superior  to  the  mixed-initiative  dialogues. 
Furthermore ,  with  the  exception  of  the  small  significant 
differences  displayed  for  different  operations,  user-  and 
computer-initiated  dialogues  led  to  similar  levels  of  search 
performance . 

The  poor  search  performance  with  the  mixed-initiative 
dialogues  was  surprising  since  the  dialogue  theoretically 
possessed  the  ability  to  improve  the  command- selection 
strategies  of  the  less  efficient  novices,  while  allowing  the 
more  proficient  novices  room  to  explore  the  advice  at  their 
own  discretion.  One  hypothesis  to  explain  the  poor 
performance  was  that  novice  users  may  not  be  ready  for  the 
additional  decision  making  introduced  by  a  mixed-initiative 
dialogue.  This  additional  level  of  decision  making  may 
result  from  the  message.  "If  you  want  advice,  press  HELP." 
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HIXED  USER  COMPUTER 


DIALOGUE  INITIATIVE 


DIALOGUE  INITIATIVE 


Figure  14.  Effect  of  dialogue  initiative  on  t  -tal  time  and 
different  operations .  Conditions  with  a  common  horizontal 
line  are  not  significantly  different ,  Newman-Keuls  ( a.  = 
0.05)  . 


127 


In  answering  this  question,  novices  would  have  to  evaluate 
their  current  progress  in  search  and  would  have  to 
anticipate  the  value  of  the  command- selection  advice.  As  a 
result  of  these  additional  mental  activities,  novices  would 
require  more  time. 

Data  were  available  to  test  the  decision-making 
hypothesis  due  to  the  complete  counterbalancing  of  the 
dialogues  in  the  aiding  phase  of  the  experiment.  The 
mixed-initiative  dialogues  were  made  available  during  each 
of  the  three  aided  sessions  (3-5)  to  different  novice 
subjects.  As  a  result,  mixed-initiative  dialogues  would  be 
available  to  novice  subjects  at  various  stages  of  search 
skill  and  familiarity  with  the  command- selection  advice. 
For  example,  a  novice  receiving  the  mixed-initiative 
dialogue  during  the  second  or  third  session  of  aiding 
probably  had  used  the  command-selection  advice  and  probably 
was  more  skilled  in  the  search  task  than  a  novice  in  the 
first  aiding  session.  With  these  knowledge  and  skills,  a 
novice  would  be  more  capable  of  dealing  with  the 
mixed-initiative  dialogues. 

Fortunately,  the  dialogue  initiative  by  presentation 
order  interaction  was  significant  for  all  dependent  measures 
indicating  that  at  least  one  of  the  dialogues  was 
differentially  effective  with  the  six  presentation  orders. 
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This  interaction  was  highly  significant  for  total  operations 
(F[10,36]  =  12.39,  p  =  0.0001),  different  operations 
( F[  10, 36  ]  =  7.31,  p  =  O.OOv'".),  total  time  (F[10,36]  =  15.09, 
p  =  0.0001),  and  total  movement  (F[10,36J  =  10.40, 

p  =  0.0001). 

Figure  15  presents  the  results  of  the  dialogue  initiative 
by  presentation  order  interaction  in  terms  of  total  time. 
The  three  other  search  measures  were  not  included  in  Figure 
15  due  to  the  similarity  of  results  for  all  search  measures 
and  the  large  amount  of  data  contained  in  this  interaction. 
In  this  figure,  the  six  possible  presentation  orders  are 
represented  by  the  sequence  of  three  letters  (i.e.,  MCU, 
MUC,  UMC,  UCM,  CMU,  or  CUM),  while  the  specific  dialogue  in 
a  presentation  order  is  denoted  by  a  single  letter  (i.e.,  C, 
M,  or  U).  For  example,  the  symbol  "CMU"  with  an  "M"  beneath 
it  would  represent  the  mixed-initiative  dialogue 
administered  in  the  second  session  of  aiding  for  the  subject 
group  that  was  presented  the  dialogues  in  a  computer-, 
mixed-,  and  user-initiated  order.  In  the  text,  this 
condition  will  be  denoted  as  CMU-M.  Included  in  Figure  15 
are  the  results  of  a  Newman-Keuls  analysis  (a  =  0.05)  for 
the  means  of  the  interaction. 

Figure  15  clearly  supports  the  decision-making 
hypothesis.  Significant  mean  differences  existed  between 
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PRESENTATION  MMCCUUCHMCUMUMUCCU 
ORDER:  UCUMMCMCUMMUMCCUUC 

CUMUCMUUCUCCCUMMMM 

DIALOGUE 

INITIATIVE:  MMCCUUMCUUCCMUCMUM 


Figure  15.  Dialogue  initiative  by  presentation  order 
interaction  for  total  time.  Conditions  with  a  common 


horizontal 

line  are  not  significantly 

different. 

Newman- 

Keuls  (a  = 

0.05) . 
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the  presentation  orders  with  mixed-initiative  condition  as 
the  first  dialogue  (MCU-M,  MUC-M)  and  all  other  combinations 
of  presentation  orders  and  dialogues.  Dialogue  presentation 
orders  with  mixed-initiative  advice  in  the  second  (UMC-M, 
CMU-M)  or  third  (CUM-M,  UCM-M)  aided  sessions  were  far 
superior  in  terms  of  total  time  than  the  mixed-initiated 
sessions  presented  in  the  first  aided  session  (MCU-M, 
MUC-M).  In  fact,  the  superiority  of  the  mixed-initiated 
dialogues  administered  during  the  second  and  third  sessions 
of  aiding  was  demonstrated  for  every  dependent  measure 
(Newman-Keuls,  a  =  0.05).  Apparently,  novice  users  were 
more  capable  of  dealing  with  the  mixed-initiative  dialogues 
after  receiving  practice  in  the  aided  task. 

The  three-way  ANOVAs  proved  to  be  useful  for  revealing  a 
surprising  sequence  effect  with  the  mixed-initiative 
dialogue.  However,  these  ANOVAs  did  not  provide  comparisons 
with  the  control  novices  and  experts.  This  comparison  data 
would  make  it  possible  to  recommend  both  effective  and 
ineffective  presentation  orders  for  the  three  dialogue 
initiatives.  Unbalanced  ANOVAs  were  conducted  for  the  four 
search  measures  to  provide  comparisons  between  the  dialogue 
presentation  orders  and  the  control  subjects  in  Experiment 
1.  The  data  were  collapsed  across  command-selection  models 
based  on  the  lack  of  any  significant  differences  (p  <  0.05) 
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in  previous  analyses  of  search  performance.  Unbalanced 
analyses  were  performed  with  the  entire  control  novice  group 
and  the  novice  group  separated  into  slow  and  fast  novices. 
Only  the  results  of  the  more  detailed  analyses  will  be 
reported  since  the  two  analyses  were  in  complete  agreement. 

The  independent  variables  for  these  unbalanced  ANOVAs 
consisted  of  the  nine  subject  groups  and  the  three  sessions 
in  which  aiding  could  occur.  The  nine  subject  groups 
included  fast  and  slow  novices/  experts,  and  the  six 
dialogue  presentation  orders  (MCU,  MUC,  UCM,  UMC,  CMU,  and 
CUM) .  All  novice  groups  had  a  sample  size  of  6,  while  the 
expert  group  had  12  subjects.  The  results  of  these 
unbalanced  ANOVAs  revealed  significant  main  effects 
(p  <  0.05)  for  subject  groups  and  sessions,  and  a 
significant  subject  group  by  session  interaction.  Both  the 
interaction  of  subject  group  and  session  and  the  main  effect 
of  subject  group  will  be  presented  to  illustrate  the 
effectiveness  of  the  six  dialogue  presentation  orders.  The 
effects  of  session  will  not  be  discussed  since  learning 
effects  will  be  displayed  in  the  presentation  of  the  subject 
group  by  session  interaction. 

The  subject  group  by  session  interaction  was  highly 
significant  for  total  operations  (£[16,102]  =  2.67, 
P  =  0.0015),  different  operations  (£[16,102]  =  2.80, 
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p  =  0.0009),  and  total  time  (F[16,102]  =  2.91,  g  =  0.0006). 
A  plot  of  the  means  for  this  interaction  in  terms  of  total 
time  is  presented  in  Figure  16.  Although  the  plot  of  the 
average  times  is  very  complicated,  visual  inspection  of  the 
results  and  Newman-Keuls  procedures  (a  =  0.05)  revealed  that 
the  dialogue  presentation  orders  of  MCU  and  MUC  were 
ineffective  in  the  first  aiding  session.  This  conclusion 
was  based  on  the  comparisons  of  the  MUC-M  and  MCU-M 
conditions  with  the  slow  novices  for  the  dependent  measures 
of  total  time,  total  operations,  and  different  operations. 
From  an  aiding  perspective,  a  dialogue  initiative  which 
leads  to  search  performance  no  better  than  the  performance 
of  slow  novices  is  not  beneficial. 

Beyond  verifying  the  ineffectiveness  of  initial 
mixed-initiative  dialogues,  Figure  16  also  suggests  that  the 
remaining  dialogue  presentation  orders  could  lead  to 
improved  performance  for  the  slower  novices,  while  not 
interfering  with  the  fast  novices.  Specifically,  the 
dialogue  presentation  orders  of  UCM,  UMC,  CMU,  and  CUM  never 
differed  significantly  from  the  fast  novices  in  total  time, 
total  operations,  or  different  operations  (Newman-Keuls, 
a  =  0.05).  Thus,  there  were  methods  of  dialogue 
presentation  that  could  facilitate  novice  performance. 
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Figure  16.  Dialogue  presentation  orders  compared  against 
slow  novices,  fast  novices,  and  experts  over  sessions . 
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The  dialogue  initiative  by  presentation  order  interaction 
provided  detailed  information  on  the  session  to  session 
effectiveness  of  different  dialogues,  but  failed  to  give  an 
overall  indication  of  which  dialogue  order  would  be  suitable 
for  further  study.  The  main  effect  of  subject  group 
supplies  this  information  by  comparing  the  dialogue 
presentation  orders  to  the  novice  and  expert  groups.  The 
search  measures  which  yielded  a  significant  subject  group 
effect  were  total  operations  (F[8,51]  =  3.05,  p  =  0.0070), 
different  operations  (F[8,51]  =  2.91,  p  =  0.0093),  and  total 
time  (F[8,51]  =  6.91,  p  =  0.0001).  The  results  of 
Newman-Keuls  procedures  (a  =  0.05)  on  the  search  means  of 
these  subject  groups  is  presented  in  Table  13. 

The  dialogue  presentation  orders  presented  in  Table  13 
that  consistently  result  in  search  performance  comparable  to 
fast  novices  and  experts  are  UCM,  CUM,  and  UMC.  Not 
surprisingly,  these  dialogue  orders  have  minimized  the 
effects  of  mixed-initiative  dialogues  since  these  dialogues 
only  occur  in  the  last  two  aiding  sessions.  Therefore,  the 
strongest  recommendation  that  can  be  made  in  terms  of 
dialogue  sequence  is  that  mixed-initiative  dialogues  should 
be  presented  only  after  users  have  had  experience  with  the 
application  task  and  the  online  advice. 
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TABLE  13 


Dialogue  Presentation  Orders  Compared  Against  Slow  novices. 
Fast  Novices,  and  Experts  Overall 


Total  Different  Total 

Operations  Operations  Time 


Mean 

Subject 

Group 

14.0 

Slow  Nov 

10.6 

MUC 

10.1 

MCU 

9.9 

CMU 

9.9 

UMC 

9.8 

UCM 

9.6 

CUM 

8.1 

Fast  Nov 

7.7 

Experts 

Subject 


Mean 

Group 

4.5 

Slow  Nov 

4.1 

MUC 

3.9 

MCU 

3.9 

UMC 

3.7 

CUM 

»  3.7 

UCM 

3.5 

Experts 

3.5 

Fast  Nov 

3.4 

CMU 

Subject 


Mean 

Group 

119.25 

Slow  Nov. 

110.59 

MUC 

105.96 

MCU 

102.83 

CMU 

93.88 

UMC 

85.21 

CUM 

82.73 

UCM 

72.65 

Fast  Nov. 

67.67 

Experts 

Means  with  a  common  vertical  line  are  not  significantly 
different,  Newman-Keuls  (a  =  0.05). 
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Interaction  with  the  online  aids.  Aided  search 
performance  indicated  that  some  novices  can  be  assisted  with 
information  retrieval  if  complicated  sequence  models  and 
highly  interactive,  mixed-initiative  dialogues  are  avoided. 
However,  to  understand  these  results  further  requires 
investigation  of  user  interaction  with  the  online  aid 

Unfortunately,  the  issue  of  how  frequently  a  user 
received  command-selection  advice  was  complicated  by  the 
implementation  of  sequence  models.  In  contrast  to  the 
frequency  and  plan-based  command- selection  advice, 
sequence-based  advice  was  allowed  several  times  a  trial. 
Thus,  an  increase  in  advice  frequency  was  expected  for  the 
sequence  models  if  advice  per  trial  was  used  as  the 
dependent  measure.  This  measure  accurately  represents 
number  of  times  advice  was  given  on  a  trial.  However,  using 
only  advice  per  trial  may  not  be  appropriate.  For  example, 
plan-based  models  offered  a  wealth  of  command-selection 
advice  that  was  not  reflected  in  the  advice  per  trial 
measure  since  plan-based  advice  could  only  be  given  once  a 
trial.  To  remedy  this  measurement  problem,  a  second  measure 
of  advice  frequency  was  developed.  This  dependent  measure 
was  the  proportion  of  advised  trials  and  was  computed  by 
counting  the  number  of  trials  for  which  command- selection 
advice  was  given.  The  total  number  of  advised  trials  was 


137 


then  converted  into  a  proportion  for  the  specific  aiding 
condition. 

The  first  set  of  analyses  performed  on  the  two  measures 
of  advice  frequency  (advice/trial  and  proportion  of  advised 
trials)  paralleled  the  session  and  model  analysis  conducted 
for  aided  search  performance.  This  analysis  attempted  to 
determine  whether  the  different  command- selection  models 
influenced  the  presentation  of  advice  over  the  three  aided 
sessions.  When  considering  advice  per  trial,  the  results  of 
the  two-way  ANOVA  found  significant  effects  for  type  of 
command-selection  model  (£[2,33]  =  12.45,  E  =  0.0001), 
session  (£[2,66]  =  4.43,  e  =  0.0156),  and  the  interaction  of 
models  by  sessions  (£[4,66]  =  3.34,  E  =  0.0150).  A  similar 
analysis  on  the  proportion  of  advised  trials  failed  tc 
detect  any  statistical  differences  (e  >  0.05)  between  models 
and  sessions. 

Advice  per  trial  is  illustrated  in  Figure  17  for  novices 
receiving  the  three  command-selection  models  across  the  3 
aided  sessions.  As  illustrated,  sequence  novices  received 
advice  frequently  during  the  first  two  aiding  sessions. 
This  high  level  of  interaction  for  the  sequence  models 
supports  an  earlier  hypothesis  that  frequent  advice  could 
have  been  responsible  for  the  poor  aided  performance  of 
sequence  novices.  However,  both  the  plot  of  the  model  by 
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session  interaction  in  Figure  17  and  the  results  of  the 
Newman-Keuls  (a  =  0.05)  procedures  showed  that  advice 
frequency  with  the  sequence  aid  decreased  to  the  level  of 
frequency  and  plan-based  models  over  the  course  of  aiding. 
The  decrease  in  presentation  frequency  was  probably  the 
result  of  two  factors.  First,  sequence  novices  were 
becoming  skilled  in  command  selection  and  may  not  have 
needed  advice  in  the  final  aiding  sessions.  Second,  in 
using  the  sequence  model,  novices  may  have  realized  the 
limitations  of  the  command- sequence  advice  and  reduced  their 
requests  for  assistance. 

The  next  set  of  analyses  on  advice  presentation  frequency 
(advice  per  trial  and  proportion  of  advised  trials) 
paralleled  the  dialogue  and  order  analyses  conducted  on 
aided  search  performance.  Dialogue  initiatives  (user, 
computer,  and  mixed),  dialogue  presentation  order,  and  the 
type  of  command-selection  model  were  analyzed  with  a 
three-way  ANOVA  to  determine  whether  advice  presentation  was 
influenced  by  the  online  dialogues  implemented  for  the 
command-selection  aids. 

Not  surprisingly,  the  results  of  these  three-way  ANOVAs 
for  type  of  command  selection  model  were  in  complete 
agreement  with  the  previous  model  by  session  analyses  on 
advice  frequency.  A  significant  main  effect  of 
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remand-selection  model  was  found  for  advice  per  trial 
(F[2,18]  =  10.48,  g  =  0.0010),  with  no  corresponding 
significant  difference  (g  <  0.05)  for  the  proportion  of 
advised  trials.  As  in  the  previous  analyses  on  models  and 
sessions,  sequence  models  provided  more  advice  (0.92 
presentations/trial)  than  frequency  models  (0.44 
presentations/trial)  and  plan-based  models  (0.38 
presentations/trial).  Newman-Keuls  comparisons  (a  =  0.05) 
verified  these  differences  in  advice  presentation  rates, 
showing  that  sequence  models  were  significantly  different 
from  both  frequency  and  plan-based  models  with  no 
statistical  difference  between  frequency  and  plan-based 
models . 

Further  outcomes  of  the  dialogue  and  order  analyses 
uncovered  large  main  effects  of  dialogue  initiative  for  both 
advice  per  trial  (F[2,36]  =  15.08,  g  =  0.0001)  and 
proportion  of  advised  trials  (F[2,36]  =  69.46,  g  =  0.0001). 
In  addition,  dialogue  initiative  was  found  to  interact  with 
type  of  command-selection  model  (F[4,36)  =  2.81,  g  =  0.0398) 
and  the  dialogue  presentation  order  (F[10,36]  =  2.41, 
g  =  0.0262)  for  the  proportion  of  advised  trials.  Both  of 
these  interactions  were  influenced  strongly  by  the  dialogue 
initiative  main  effect,  and  as  a  result  will  not  be 
presented. 
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The  means  and  the  results  of  Newman-Keuls  procedures 
(a  =  0.05)  for  the  three  dialogue  initiatives  are  presented 
in  Figure  18.  Both  dependent  measures  revealed  significant 
increases  for  advice  presentation  with  the 
computer-initiated  dialogues.  Figure  18  also  illustrates 
that  requests  for  advice  did  not  increase  from  the 
user-initiated  to  the  mixed-initiated  dialogues.  In  fact, 
if  both  search  performance  and  advice  frequency  are 
considered,  mixed-initiative  dialogues  appear  to  be 
relatively  ineffective  in  changing  novice  search  behavior. 

Transfer  search  performance .  The  results  described  in 
this  section  focus  on  the  performance  of  the  aided  subjects 
during  the  final  session  of  the  experiment.  In  this  final 
session,  aided  novices  retrieved  information  without  the 
online  aid.  The  purpose  of  this  analysis  was  to  determine 
if  novices  acquired  any  skills  through  the  use  of  the  online 
aids  that  facilitated  search  performance  above  and  beyond 
mere  practice.  Several  questions  were  asked  in  this 
transfer  analysis.  First,  were  aided  novices  finding 
information  faster  and  more  efficiently  than  the  control 
novices  in  Experiment  1?  Second,  if  aided  novices  were 
finding  information  faster,  then  were  these  aided  novices 
performing  at  an  expert's  level  of  search  proficiency? 
Finally,  did  the  online  aiding  lead  to  any  improvements  in 
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F-l-sure  18.  Advice  presentation  frequency  for  dialogue 
initiatives .  Conditions  with  a  common  horizontal  line  are 
not  significantly  different,  Newman-Keuls  (a  =  0.05). 
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the  variability  of  search  performance  for  the  novice  subject 
groups? 

The  first  analysis  conducted  was  an  overall  transfer 
analysis  on  the  aided  novices,  control  novices,  and  experts 
for  total  operations,  different  operations,  total  time,  and 
total  movement  in  the  last  retrieval  session.  The  results 
of  the  one-way  MANOVA  revealed  no  overall  search  performance 
differences  between  the  aided  novices,  control  novices,  and 
experts,  Wilks’  A(4,55)  =  0.6544,  p  =  0.1114.  From  this 
overall  analysis  it  could  be  concluded  that  search 
performance  was  not  changed  as  a  result  of  the  online 
aiding.  However,  the  heterogeneity  of  novice  search 
performance  demonstrated  in  Experiment  1  may  have  clouded 
these  results.  To  investigate  this  hypothesis,  control 
novices  were  separated  into  groups  previously  described  as 
slow  and  fast  novices.  Overall  transfer  performance  was 
then  reanalyzed  for  the  six  subject  groups  in  a  MANOVA  model 
with  unbalanced  sample  sizes.  Once  again,  the  assumption 
with  the  separated  novice  groups  was  that  both  fast  and  slow 
novices  would  be  included  in  the  groups  of  aided  novices. 
The  results  of  this  MANOVA  indicated  that  there  were  overall 
search  performance  differences  between  aided  novices,  fast 
and  slow  novices,  and  experts,  Wilks'  A(5,54)  =  0.4437, 

p  =  0.0016. 
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Due  to  the  overall  search  differences  between  subject 
groups,  detailed  unbalanced  ANOVAs  on  each  dependent  measure 
were  conducted.  Statistical  differences  between  the  six 
subject  groups  were  found  for  total  operations 
(F[5,54]  =  4.84,  g  =  0.0011),  different  operations 
(F[5, 54]  =  3.05,  g  =  0.0169),  total  time  (F[5,54]  =  7.14, 
g  =  0.0001),  and  total  movement  (F{5,54]  =  4.11, 
g  =  0.0032).  The  search  means  for  the  six  subject  groups 
are  displayed  in  Figure  19.  Included  in  Figure  19  are  the 
results  of  the  Newman-Keuls  procedures  (o  =  0.05). 

The  large  and  consistent  differences  shown  in  Figure  19 
strongly  indicated  that  aided  novice  subjects  were 
retrieving  information  faster  and  more  efficiently  than  the 
slower  novices.  Aided  novices,  when  compared  to  slow 
novices,  retrieved  information  approximately  60%  faster, 
used  60%  fewer  .otal  search  operations,  were  50%  more 
accurate  in  terms  of  line  movement,  and  constrained  their 
use  of  search  procedures  by  75%.  In  fact,  during  transfer 
aided  novices  were  retrieving  information  just  as 
efficiently  as  the  fast  novices  and  experts  irrespective  of 
the  type  of  command- selection  advice  administered  in  the 
three  previous  sessions.  All  three  command-selection  aids 
appear  to  have  improved  the  search  performance  of  the  slower 
novices  without  impairing  the  faster  novices. 


SN  FN  EX  FRQ  SEQ  PIN  SN  FN  EX  FRQ  SEQ  PLN 


different,  Newman-Keuls  (a  =  0.05). 
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Closely  associated  with  the  issue  of  average  search 
performance  in  transfer  was  the  question  of  whether  aided 
novices  became  more  homogeneous  in  search  performance.  That 
is,  did  the  command- selection  aids  reduce  the  large 
variability  in  novice  search  performance?  To  answer  this 
question,  a  Moses  rank-like  test  (Hollander  and  Wolfe,  1973) 
was  conducted  for  total  operations  and  total  time.  Only 
total  operations  and  total  time  were  focused  on  in  transfer 
since  these  were  the  search  measures  which  demonstrated 
significant  differences  (p  <  0.05)  in  variability  of  control 
novices  and  experts  in  Experiment  1. 

The  first  two  Moses  rank-like  tests  were  performed  on  the 
dispersion  differences  between  control  novices  and  experts 
in  the  transfer  session.  The  results  of  these  tests 
confirmed  the  dispersion  differences  between  control  novices 
and  experts  in  the  transfer  session  for  total  time 
(W[4,4]  =  26.0,  p  =  0.014),  but  failed  to  detect  dispersion 
differences  for  total  operations  (W[4,4J  =  23.0,  p  =  0.1). 
Therefore,  comparisons  of  the  variability  in  transfer 
performance  between  the  three  aided  groups,  and  the  novices 
and  experts  in  Experiment  1  only  focused  on  total  time.  Box 
plots  of  this  transfer  data  for  all  five  subject  groups  are 
p^own  in  Figure  20. 


TOTAL  TIME 


Figure  20.  Total  times  for  aided  novices,  control  novices 
and  exDerts  in  the  transfer  session. 


148 


Figure  20  shows  the  differences  in  search  time  dispersion 
for  the  five  subject  groups.  The  online  aiding  appeared  to 
decrease  the  variability  within  the  novice  groups. 
Estimates  of  group  variances  and  detailed  comparisons  of  the 
aided  novice  groups  with  the  novices  and  experts  in 
Experiment  1  are  provided  in  Table  14.  As  shown  in  Table 
14,  the  variability  of  each  aided  group  was  compared 
individually  with  the  control  novices  and  experts  using  the 
Moses  rank-like  test.  With  the  exception  of  the  frequency 
novices,  online  aiding  decreased  variability  in  transfer 
times  so  that  aided  novices  groups  were  beginning  to 
resemble  experts  in  terms  of  group  variation.  Together  with 
the  demonstrated  differences  between  slow  novices  and  aided 
novices,  these  results  suggest  that  command- selection  aids 
improved  the  performance  of  slower  novices  thereby  making 
group  performance  more  consistent. 

Search  strategy.  The  improvements  in  performance  and 
transfer  lead  to  the  question  of  whether  aided  novices  had 
changed  their  search  strategies  during  the  course  of  the 
experiment.  The  results  of  Experiment  1  found  that  the 
slower  novices  tended  to  use  the  SCROLL  UP,  SCROLL  DOWN,  and 
ZOOM  OUT  more  frequently  than  the  faster  novices  and 
experts.  Therefore,  the  goal  of  the  search  strategy 
analysis  was  to  determine  whether  novices  decreased  the  use 
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TABLE  14 


Comparisons  of  the  Variability  of  Total  Time  in  Transfer  for 
Aided  Novices,  Control  Novices,  and  Experts. 


Experiment  1  Subject  Groups 


Experiment  2 

Novices 
o2  =  1046.60 

Experts 
a2  =  110.35 

Subject  Groups 

Frequency  Novices 

20.0 

25.0 

a2  =  338.74 

(0.343) 

(0.029) 

Sequence  Novices 

24.0 

20.0 

a2  =  108.74 

(0.057) 

(0.343) 

Plan  Novices 

22.0 

24.0 

a2  =  235.49 

(0.171) 

(0.057) 

Main  entries  are  rank  sum  statistics  W(4,4). 
Values  in  parentheses  represent  g-values. 
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of  these  search  procedures  as  a  result  of  having  online 
command-selection  assistance. 

The  analysis  of  the  search  strategy  paralleled  the 
analysis  conducted  in  Experiment  1.  In  this  experiment, 
however,  strategy  analysis  was  conducted  immediately  at  a 
detailed  level  comparing  frequency  novices  (12  subjects), 
sequence  novices  (12  subjects),  plan  novices  (12  subjects), 
slow  novices  (6  subjects),  fast  novices  (6  subjects),  and 
experts  (12  subjects).  The  proportions  of  search  procedure 
selection  for  each  subject  group  were  analyzed  with  a  MANOVA 
to  check  for  overall  differences  in  search  strategies.  The 
results  of  this  one-way  MANOVA  revealed  a  significant  Wilks' 
criterion,  A[5,54]  =  0.1607,  p  =  0.0063,  indicating  that 
some  combination  of  these  search  procedures  were  selected 
differently  between  subject  groups.  Individual  unbalanced 
ANOVAs  were  conducted  subsequently  to  investigate  these 
differences  in  search  procedure  selection.  The  results  of 
these  ANOVAs  showed  significant  differences  in  the  selection 
of  SCROLL  UP  (FI  5, 54]  =  2.41,  p  =  0.0480),  SCROLL  DOWN 
(F[  5, 54]  =  2.44,  p  =  0.0455),  and  ZOOM  OUT  (F[  5 , 54]  =  5.10, 
p  =  0.0007)  for  the  six  subject  groups.  Figure  21  displays 
the  selection  propprtions  of  these  three  search  procedures 
for  each  of  the  six  subject  groups.  Included  in  Figure  21 
are  the  results  of  Newman-Keuls  procedures  (a  =  0.05)  on 
these  average  proportions. 
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GROUP:  SN  PLN  SEQ  FRQ  EX  FN  SN  PLN  SEQ  FRQ  EX  FN  SNSEQ  FRQ  PLN  EX  FN 

SCROLL  UP  SCROLL  DOWN  ZOOM  OUT 


Figure  21.  Significant  di f f erences  in  search  procedure 
selection  for  aided  novices  ( frequency  -  FRQ,  sequence  - 
SEQ,  plan  -  PLN) ,  slow  novices  (SN),  fast  novices  (FN),  and 
experts  (EX).  Groups  with  a  common  horizontal  line  are  not 
significantly  different,  Newman-Keuls  (a  =  0.05)T 
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Figure  21  demonstrates  the  decrease  in  the  selection  of 
the  SCROLL  UP  and  ZOOM  OUT  by  novices  receiving 
command-selection  advice.  Aided  novices  were  selecting 
these  search  procedures  less  often  than  the  slow  novices  and 
were  approaching  the  selection  rates  of  expert  subjects. 
The  reduction  in  the  selection  of  SCROLL  DOWN  was  not  as 
large.  Aided  novices  decreased  the  use  of  the  SCROLL  DOWN 
to  a  level  comparable  to  experts  and  fast  novices,  but  this 
reduction  was  not  large  enough  to  lead  to  significant 
differences  between  aided  and  slow  novices. 

As  a  result  of  receiving  command-selection  advice  during 
the  experiment,  aided  novices  were  gaining  command-selection 
skills  in  the  experiment  and  some  knowledge  of  the  file 
hierarchy.  Diminished  use  of  the  scrolling  procedures  was 
indicative  of  the  acquisition  of  command- selection  skill  by 
aidpd  novices  since  other  procedures  like  SEARCH  or  INDEX 
can  typically  position  a  file  quickly  to  a  desired  position. 
Likewise,  the  decreased  use  of  ZOOM  OUT  was  an  indication  of 
aided  subjects  learning  the  structure  of  the  file  hierarchy. 
The  ZOOM  OUT  procedure  is  only  used  when  subjects  are  in  the 
wrong  file.  The  different  command-selection  models  did  not 
lead  to  any  significantly  different  search  strategies. 
Detailed  statistical  procedures  may  detect  these  differences 
during  aiding  or  transfer,  however,  these  analyses  were  not 
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conducted  due  to  small  sample  sizes  associated  with  search 
problems  and  subjects. 

Satisfaction  with  the  online  aids.  Subjective  responses 
to  the  command-selection  aids  and  dialogue  initiatives  were 
measured  through  a  set  of  bipolar  adjective  scales 
administered  at  the  end  of  each  aiding  session  and  a 
preference  ranking  of  the  dialogue  initiatives  at  the  end  of 
the  experiment.  These  two  dependent  measures  were  analyzed 
separately.  The  purpose  of  these  analyses  was  to  determine 
which  type  of  online  command-selection  aid  and  apologue 
initiative  was  most  desirable  from  the  user's  perspective. 

The  bipolar  adjective  scales  were  analyzed  within  an 
ANOVA  model.  Parametric  procedures  were  selected  over 
nonparametric  since  the  12  bipolar  scales  provided  for  a 
multi-variable  analysis.  In  this  situation,  parametric 
tests  are  well  suited  for  protecting  against  the  inflation 
of  alpha  error.  In  addition,  the  analysis  of  bipolar  scales 
was  complicated  due  to  missing  data  points.  Three  novices 
with  user-initiated  dialogues  (two  frequency  novices  and  one 
sequence  novice)  did  not  request  advice  from  the  online  aid. 
These  subjects  could  not  be  expected  and  were  not  asked  to 
evaluate  the  online  aid  for  those  specific  conditions. 
Therefore,  unbalanced  ANOVAs  were  employed  using  the  method 
of  weighted  means  to  estimate  the  missing  values  (Winer, 
1971) . 
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The  analysis  of  the  bipolar  scales  was  separated  into 
three  parts.  The  first  analysis  was  a  two-way  ANOVA  on  the 
overall  satisfaction  scale.  The  two  factors  included  in 
this  unbalanced  ANOVA  were  type  of  command-selection  model 
and  dialogue  initiative.  The  results  of  this  ANOVA  did  not 
find  any  statistically  reliable  (p  <  0.05)  differences  in 
user  satisfaction  that  could  be  attributed  to  the 
command-selection  model  or  the  dialogue  initiative. 

The  analysis  then  proceeded  to  the  individual  bipolar 
adjective  scales.  These  scales  were  analyzed  in  two  groups. 
The  first  set  of  bipolar  scales  analyzed  were  part  of  the 
evaluation  instrument  developed  and  validated  by  Coleman 
(1985).  These  scales  were  the  first  8  individual  bipolar 
adjectives  displayed  in  Table  11.  These  8  bipolar  scales 
were  separated  from  the  other  3  individual  scales  to  avoid 
confounding  the  results  of  a  previously  developed  and  tested 
evaluation  tool  with  other  less  well  selected  and  validated 
scales  (bipolar  adjective  scales  10  -  12). 

To  safeguard  the  alpha  error,  a  single  ANOVA  was 
conducted  on  the  8  individual  scales  from  Coleman's  (1985) 
user  evaluation  instrument.  Rating  scale  in  this  ANOVA  was 
treated  as  a  within-subjects,  classification  variable  and 
was  used  to  determine  if  the  scale  ratings  were  influenced 
by  either  the  command-selection  model  or  the  dialogue 
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initiative.  However,  due  to  computational  limitations, 
missing  values  in  this  ANOVA  had  to  be  eliminated.  For  this 
purpose,  means  were  computed  across  the  replicated  and 
counterbalanced  dialogue  orders  reducing  the  cell  size  by 
1/2  and  effectively  eliminating  any  missing  values.  The 
results  of  this  reduced  ANOVA  revealed  a  significant  scale 
by  model  interaction  (F[14,105]  =  1.87,  p  =  0.0380)  for  the 
8  individual  bipolar  scales. 

To  further  investigate  the  scale  by  model  interaction  for 
these  8  bipolar  scales,  8  individual  unbalanced  ANOVAs  were 
conducted.  The  results  of  these  ANOVAs  yielded 
statistically  different  ratings  of  the  command-selection 
r'dels  for  the  DISGUSTING-PLEASING  bipolar  adjectives 
(FI  2, 33]  =  3.37,  p  =  0.0465)  and  the  UNDEPENDABLE-DEPENDABLE 
bipolar  adjectives  ( F [ 2 , 33  ]  =  4.79,  p  =  0.0149)  The  mean 
ratings  for  these  two  bipolar  scales  are  presented  in  Figure 
22  along  with  the  results  of  the  Newman-Keuls  comparisons 
(a  =  0.05)  on  these  means. 

The  bar  charts  in  Figure  22  illustrate  the  more  favorable 
response  given  to  the  plan-based  models.  The  novices  in  the 
plan-based  group  rated  the  assistance  as  slightly  pleasing 
and  slightly  dependable.  These  positive  ratings  of  the 
plan-based  advice  are  to  be  contrasted  with  the  sequence 
advice  which  was  rated  halfway  between  neutral  and  slightly 
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disgusting  or  slightly  undependable.  Thus,  the  simple  and 
unchanging  sequence  of  search  procedures  supplemented  with 
verbal  explanations  may  be  necessary  ingredients  for 
improving  the  subjective  quality  of  command- selection 
advice . 

The  second  set  of  bipolar  adjectives 
( UNADAPT I VE- ADAPTIVE,  STUPID-INTELLIGENT,  and 
HINDERING-HELPFUL)  were  analyzed  similarly.  An  initial 
balanced  ANOVA  based  on  the  mean  ratings  of  subjects  from 
replicated  dialogue  orders  was  conducted  to  safeguard  alpha 
error.  The  three-way  ANOVA  included  scale  as  a  factor  to 
determine  if  ratings  on  individual  bipolar  adjectives  were 
influenced  by  the  command-selection  model  or  the  dialogue 
initiative.  The  results  of  the  ANOVA  revealed  no 
significant  interactions  with  scale  (p  <  0.05).  Further 
analyses  were  abandoned  and  the  results  accepted  as 
indicating  the  insensitivity  of  these  intuitively  selected 
scales . 

The  final  subjective  analysis  was  a  nonparametric  test  of 
the  preference  rankings  for  the  three  dialogue  initiatives. 
A  one-way,  blocked  Friedman  rank  sum  test  was  used  for  each 
aided  group  (frequency,  sequence,  and  plans)  to  determine 
which  dialogue  initiative  was  preferred.  Significant 
differences  in  preference  for  the  three  dialogues  were  found 
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with  the  frequency  model  (S'  =12.5,  g  <  0.005),  and  with 
the  sequence  model  (S'  =  8.17,  g  <  0.025).  No  significant 
differences  in  preference  were  found  for  dialogue 
initiatives  with  the  plan-based  model  (S'  =  2.0,  g  >  0.10). 
Individual  rank  sums  of  the  three  dialogue  initiative  for 
each  command- selection  model  are  provided  in  Figure  23. 
Included  in  Figure  23  are  the  results  of  the  multiple 
comparisons  on  the  individual  rank  sums  (a  =  0.05)  for  the 
frequency  and  sequence  groups  (Hollander  and  Wolfe,  1973). 
Care  should  be  taken  when  interpreting  Figure  23  since  a 
large  rank  sum  preference  indicates  a  lesser  preferred 
dialogue . 

Figure  23  displays  the  lack  of  preference  for  the 
computer-initiated  advice  in  the  frequency  and  sequence 
models.  Computer-initiated  dialogues  with  these 
command-selection  models  could  be  extremely  intrusive  to  a 
user's  search  strategy.  Subjects  could  be  interrupted  and 
provided  command- selection  advice  without  any  supporting 
explanations  for  the  search  procedures  suggested.  This 
intrusion  hypothesis  also  was  supported  by  the  lack  of 
significant  preference  differences  for  the  plan-based  aid. 
Computer-initiated  dialogues  with  the  plan-based  model 
provides  information  on  how  to  use  the  command-selection 
advice.  Therefore,  the  lack  of  a  preference  difference 
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between  dialogues  may  indicate  a  willingness  on  the  part  of 
some  novice  subjects  to  use  the  computer-initiated, 
plan-based  advice. 

Summary 

The  command-selection  aids  were  capable  of  assisting 
slower  novices  with  information  retrieval  and  were  suitable 
for  training  these  novices  on  efficient  search  strategies 
resulting  in  a  significant  improvement  in  transfer  search 
performance.  In  terms  of  group  performance,  these  changes 
led  to  a  decrease  in  the  heterogeneity  of  novice  search 
performance.  However,  the  experiment  also  revealed  that  the 
command-selection  models  and  the  dialogue-initiatives  must 
oe  designed  carefully  to  avoid  providing  complicated  advice 
and  to  avoid  an  initial  highly  interactive  dialogue. 


GENERAL  DISCUSSION 


Command-Selection  Models 

Why  were  the  online  aids  successful  in  providing  advice 
to  the  slower  novices?  The  answer  may  reside  in  the 
construction  of  command- selection  models  based  on  the  search 
clusters.  Development  of  the  search  clusters  defined  a  set 
of  component  search  tasks  for  which  consistent 
command- selection  advice  could  be  given  to  the  novice  user. 
That  is,  novice  users  would  be  presented,  at  the  very  least, 
the  same  set  of  search  procedures  for  similar  types  of 
search  problems.  This  provides  the  novice  a  chance  to 
practice  and  to  process  the  command- selection  information 
consistently  over  several  trials.  With  consistent  practice, 
the  training  principles  of  Schneider  (1985)  suggest  that 
users  will  be  more  likely  to  learn  and  possibly  automate  the 
necessary  search  skills. 

Improvements  as  a  result  of  consistent  practice  are  best 
supported  by  the  performance  of  the  novices  receiving  the 
plan-based  models.  With  the  plan-based  models  novice  users 
would  be  provided  the  same  set  of  search  procedures  as 
advice  for  similar  types  of  search  problems.  Moreover,  in 
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the  search  plan  explanations  subjects  would  be  told 
explicitly  what  aspects  of  the  search  problem  to  look  for 
and  how  the  search  procedures  should  be  used.  The  novice 
user  was  provided  very  consistent  advice  related  to  the 
search  problem  and  should  be  expected  to  learn  the  search 
task.  In  fact,  subjective  evaluations  of  the  plan-based 
model  were  indicative  of  the  pleasing  and  dependable  advice 
administered  when  compared  to  the  sequence  model. 

Unfortunately,  consistent  practice  can  only  play  a  small 
role  with  frequency  and  sequence  models  since  the  component 
search  tasks  were  not  explicitly  presented.  However, 
another  related  theory,  that  of  learning  by  active  learning 
(Carroll  and  Mack,  1984;  1985),  may  be  useful  in 
understanding  the  improved  transfer  performance  of  novices 
receiving  frequency  and  sequence  advice. 

With  frequency  and  sequence  advice,  novices  were 
provided,  by  definition,  an  incomplete  model  (Carroll  and 
Mack,  1985)  of  command  selection.  Verbal  explanations  for 
using  the  suggested  search  procedures  were  not  provided  with 
the  frequency  and  sequence  advice.  In  order  to  use  the 
frequency  and  sequence  advice  novices  would  have  to  explore 
actively  the  search  procedures.  Therefore,  as  a  result  of 
learning  by  doing  (Carroll  and  Mack,  1984),  and  possible 
self-elaborations  (Mayer,  1981),  novice  subjects  would 
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acquire  search  strategies  that  would  facilitate  improved 
transfer  performance.  The  improved  transfer  search 
performance  of  frequency  and  sequence  novices  over  the  slow 
novices  suggests  that  a  more  active  learning  environment 
existed  with  these  command-selection  aids. 

There  may  be  many  other  reasons  why  the  command- selection 
models  were  successful.  However,  one  of  the  most  important 
reasons  that  should  not  be  overlooked  is  feedback.  Eberts 
and  Brock  (1984)  in  reviewing  computer-based  instruction  and 
Wiliiges,  Wiliiges,  and  Elkerton  (in  press)  in  reviewing 
software  interface  design  have  both  noted  the  importance  of 
feedback  as  a  design  principle.  A  human-computer  interface 
should  be  closed-loop  with  feedback  to  the  user  about 
quality  of  their  performance  and  the  steps  necessary  to 
cause  some  desired  action. 

Clearly,  novices  were  provided  additional  feedback  with 
these  command- selection  models  that  the  slower  novices  did 
not  have.  Similar  to  the  interpretation  given  by  Carroll 
and  Carrithers  (1984),  the  feedback  given  with  the 
command- selection  advice  limited  the  number  of  search 
commands  allowing  novices  to  identify  and  correct 
inefficient  search  strategies  and  improve  transfer 
performance.  Still,  feedback  must  be  applied  appropriately. 
Too  much,  or  inappropriate  feedback,  as  was  given  with  the 
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sequence  models,  may  degrade  aided  performance  and  should  be 
avoided  in  computer-based  learning  environments  (Robinson 
and  Knirk,  1984). 

As  a  final  word  on  command-selection  models,  the 
interpretation  of  success  must  be  considered.  Success  in 
this  experiment  meant  that  extremely  slow  and  inefficient 
novice  search  performance  was  not  observed  with  the 
command-selection  aids.  If  success  is  measured  by  overall 
mean  differences  of  groups,  then  the  merits  of  these 
command- selection  aids  are  questionable.  How  might  the  lack 
of  overall  differences  be  explained? 

First,  the  question  of  model  optimization  must  be  asked. 
That  is,  was  each  of  the  command-selection  models  the  best 
possible  behavioral  implementation?  Despite  the  time, 
effort,  and  thought  put  into  designing  the  command-selection 
models,  there  were  indications  that  improvements  could  be 
made.  For  example,  the  problem  of  complicated  and  frequent 
advice  was  demonstrated  for  the  sequence  model.  Methods  and 
procedures  for  limiting  the  advice  might  result  in  making 
the  sequence  model  viable.  Similarly,  due  to  the  cognitive 
nature  of  the  search  plans  there  was  probably  room  for 
improvement  either  through  iterative  design  procedures 
(Sullivan  and  Chapanis,  1983)  or  graphic  presentations 
(Sebrechts,  Deck,  and  Black,  1983).  Thus,  conclusions  on 
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the  relative  effectiveness  of  the  command- selection  models 
must  be  interpreted  carefully  since  a  slightly  different 
implementation  could  change  the  results  and  possibly  lead  to 
the  superiority  of  a  specific  model  for  aiding  novice  users. 

Related  to  the  issue  of  optimization  was  the  experimental 
design  used  in  this  study.  In  order  to  investigate  dialogue 
initiatives,  aided  novices  were  given  several  dialogues  to 
interact  with  the  online  aid.  The  lack  of  a  consistent 
dialogue  could  have  dramatically  affected  search  performance 
and  eliminated  any  differences  between  models.  A 
between- subjects  design  with  dialogues  is  recommended  in 
future  research.  Therefore,  more  research  is  necessary  to 
further  understand  many  of  these  modeling  issues. 

Dialogue  Initiatives 

As  shown  in  this  experiment,  the  relative  success  of  the 
online  command-selection  aids  was  also  influenced  by  the 
dialogue  initiative.  In  fact,  the  most  surprising  result 
was  the  poor  performance  of  aided  subjects  receiving 
mixed-initiative  advice  early  in  the  experiment.  Aided 
subjects  did  not  seem  to  be  ready  for  the  additional 
decision  making  required  to  determine  whether  advice  should 
be  presented  or  not.  The  conclusion  reached  in  this 
experiment  was  that  the  highly  interactive  mixed-initiative 
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dialogue  should  be  presented  only  when  the  novice  becomes 
familiar  both  with  the  task  and  the  command-selection  advice 
to  avoid  any  performance  decrements.  In  online  training, 
even  temporary  performance  decrements  may  rob  the  learner  of 
additional  trials  and  cancel  any  time  compression  benefits 
thought  to  be  crucial  to  the  development  of  cognitive  and 
high-performance  skills  (Eberts  and  Brock,  1984;  Schneider, 
1985) . 

This  conclusion  coincides  with  principles  advanced  by 
Robinson  and  Knirk  (1984)  for  presenting  instructional 
assistance  in  training  interfaces.  Robinson  and  Knirk 
(1984)  state  that  instructional  aids  should  not  add 
unnecessary  dialogue  to  training.  Unfortunately,  the 
mixed-initiative  dialogues  introduced  an  implied  dialogue 
when  suggesting  that  command-selection  advice  was  available. 
Novices  were  having  to  make  additional  decisions  about 
receiving  command-selection  advice.  Similar  to  the  results 
found  by  Carroll  and  Kay  (1985)  with  scenario  machines,  the 
coherence  of  the  task  may  have  been  destroyed  by  the 
additional  prompts  and  feedback  in  the  training  dialogue. 

Finally,  when  implementing  dialogues,  the  preferences  of 
users  should  be  considered.  From  this  investigation,  it  is 
clear  that  unless  the  command-selection  aid  provides  verbal 
explanations  for  using  the  advice  (i.e.,  plan-based  advice). 
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users  will  not  like  to  have  the  computer  determine  when 
advice  should  be  given.  Therefore,  user-initiated  dialogues 
are  recommended  for  the  command-selection  assistance 
presented  in  this  experiment  due  to  the  performance 
limitations  of  the  mixed-initiative  dialogues  and  the 
preference  limitations  of  the  computer-initiated  dialogues. 
This  recommendation  is  consistent  with  the  guidelines  for 
human- computer  interfaces  (Williges  and  Williges,  1984)  and 
the  online  help  research  of  Cohill  and  Williges  (1985). 
However,  further  research  should  be  conducted  to  determine 
if  computer-  and  mixed-initiative  dialogues  can  be  improved 
to  encourage  more  active  exploration  and  browsing  (Carroll 
and  Mack,  1984;  Cohill  and  Williges,  1985)  of  the  software 
interface  by  inexperienced  users. 

Information  Retrieval  Skills 

A  final  noteworthy  result  of  these  experiments  was  the 
wide  ranging  skills  of  novice  subjects  learning  to  use  the 
information  retrieval  interface.  Some  novices,  after 
initial  training,  were  just  as  skilled  as  more  experienced 
subjects.  Other  novices  were  dreadfully  slow  in  learning 
how  to  use  the  information  retrieval  system.  The  large 
differences  in  search  performance  existed  even  though  all 
subjects  had  received  one  tutorial  session  on  the  database 
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and  search  procedures,  and  one  session  of  practice  on  the 
actual  task.  From  these  results,  the  conclusion  was  drawn 
that  novice  performance  resembled  the  acquisition  of 
high-performance  skills.  However,  the  match  between 
information  retrieval  skills  in  the  experiment  and  the 
high-performance  skills  described  by  Schneider  (1985)  was 
not  perfect.  Schneider  (1985)  stated  that  high-performance 
skills  may  take  100  hours  or  more  to  learn  and  may  lead  to 
substantial  failure  rates  in  learners.  Clearly,  information 
retrieval  in  this  experiment  did  not  approach  this  level  of 
difficulty. 

Therefore,  additional  questions  must  be  asked  surrounding 
the  heterogeneity  of  novice  search  performance  to  gain  a 
fuller  understanding  of  the  diverse  skills  of  novices  in  the 
information  retrieval  task.  Some  of  these  questions  might 
be:  How  robust  is  this  phenomenon?  Was  the  effect 
introduced  by  the  training  procedures?  Does  the  effect 
magnify  without  training?  Can  novice  skills  be  predicted? 
Until  further  research  answers  some  of  these  questions, 
attempts  at  aiding  novices  will  be  limited. 

Nevertheless,  these  large  individual  differences,  or  what 
has  been  referred  to  as  heterogeneity  of  search  performance, 
provided  a  unique  opportunity  to  investigate  skill 
differences  in  the  laboratory.  in  fact,  from  this 
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investigation,  and  other  previous  investigations  (Elkerton 
and  Williges,  1984;  1985),  a  few  reasonable  hypotheses  can 
be  formulated  as  to  why  heterogeneity  of  novice  search 
performance  was  observed.  In  previous  investigations 
(Elkerton  and  Williges,  1984;  1985)  novices  were  only 
required  to  search  one  file  at  a  time.  In  the  present 
investigation  subjects  were  asked  to  search  through  several 
files  in  a  hierarchy.  As  a  result,  data  organization  was 
much  more  complicated.  In  addition  to  having  to  know  where 
the  information  was  in  a  file,  novices  had  to  know  where 
this  file  was  in  the  hierarchy. 

Why  might  the  complicated  organization  lead  to  widely 
varying  search  performance?  The  answer  may  be  in  the 
conceptual  structure  that  novices  ascribe  to  the  database. 
The  database  in  the  first  two  studies  could  be  conceived  of 
as  a  linear  list.  Both  investigations  (Elkerton  and 
Williges,  1984;  1985)  have  'amonstrated  that  novices  treat 
the  database  as  a  list  due  to  the  large  amount  of  scrolling. 
In  the  present  study  this  was  also  true  of  the  slower 
novices.  However,  the  scrolling  behavior  was  more  damaging 
to  performance  with  a  file  hierarchy  due  to  the  large  amount 
of  data  that  could  be  scrolled  through  (2780  lines).  In 
addition,  the  file  hierarchy  introduces  another 
organizational  structure  which  novices  have  to  master.  The 
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use  of  ZOOM  OUT  by  slower  novices  in  the  present 
investigation  was  indicative  of  the  problems  they  were 
having  with  the  file  hierarchy.  Consequently,  the 
performance  of  slow  novices  was  compounded  by  not  knowing  in 
which  file  the  information  is  located. 

However,  the  additional  hierarchical  data  structure  does 
not  impede  the  performance  of  every  novice  subject.  Novices 
have  the  capability  to  understand  the  organization  of  the 
file  hierarchy.  Indeed,  Durding,  Becker,  and  Gould  (1977) 
have  illustrated  the  capability  of  individuals  to 
conceptualize  and  use  several  data  structures  like  lists, 
networks,  tables,  and  hierarchies  to  organize  data.  But  as 
Durding,  Becker,  and  Gould  (1977)  also  point  out, 
individuals  sometimes  rely  entirely  on  lists  to  organize 
data.  Thus,  the  large  variability  in  search  performance  may 
be  influenced  by  the  conceptual  organization  created  by 
novices  when  learning  the  information  retrieval  task.  The 
fast  novices,  like  experts,  identified  the  hierarchy  quickly 
and  used  this  knowledge  to  organize  search.  Slower  novices 
may  not  have  fully  understood  the  hierarchical  organization 
leading  to  difficulties  in  locating  the  correct  file  for 
information  retrieval. 

However,  the  conceptual  organization  hypothesis  is  one  of 
several  hypotheses  that  need  to  be  tested.  Several  factors 
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may  have  contributed  to  the  heterogeneity  of  novice  search 
performance.  For  example,  some  novices  in  attempting  to 
manipulate  all  the  information  required  for  solution  of  the 
search  problem  may  have  been  limited  by  spatial  memory 
constraints  or  their  analytical  ability.  Further  research 
will  be  required  to  identify  both  the  characteristics  of  the 
task  and  the  characteristics  of  the  users  which  contribute 
to  the  variability  in  novice  search  performance. 


CONCLUSIONS 


The  results  of  these  experiments  demonstrated  that  the 
widely  varying  search  performance  of  novice  subjects  could 
be  improved  through  the  use  of  online  command-selection 
aids.  The  search  commands  provided  as  advice  by  the  online 
aids  provided  an  active  and  feedback-rich  environment  where 
novices  could  learn  and  practice  the  information  retrieval 
task.  With  these  suggested  search  procedures  slower  novices 
were  able  to  identify  and  correct  inefficient  scrolling 
strategies  permitting  these  subjects  to  learn  and  explore 
the  information  retrieval  database. 

The  different  types  of  command- selection  advice  presented 
by  the  frequency,  sequence,  and  plan-based  models  had  little 
effect  on  search  performance.  Frequency,  sequence,  and 
plan-based  models  were  all  equally  effective  as  a  training 
device.  Sequence  models,  however,  demonstrated  some 
difficulties  with  assisting  novices  due  to  the  complicated 
and  frequent  advice  offered  to  the  subject.  Subjective 
evaluations  verified  these  difficulties  since  novices 
receiving  the  command- sequence  advice  evaluated  the  online 
aid  less  favorably  than  novices  receiving  the  plan-based 
advice . 
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In  providing  this  command- selection  advice, 
mixed-initiative  dialogues  were  found  to  result  in  poor 
search  performance  in  the  first  session  of  aiding.  The  poor 
search  performance  was  only  temporary  with  mixed-initiative 
dialogues,  but  large  enough  to  suggest  careful 
implementation  of  these  dialogues  when  introducing  a 
software  interface  to  a  new  user.  Moreover,  the  transient 
nature  of  the  poor  search  performance  indicated  that  the 
effect  may  be  due  to  the  self-evaluation  required  by  the 
user  when  deciding  whether  to  receive  the  suggested  advice. 

These  effects  of  mixed-initiative  dialogues  were  not 
reflected  in  the  user  preference  rankings  of  the  three 
dialogues.  Rather,  a  strong  lack  of  preference  for  the 
computer-initiated  dialogues  was  demonstrated  with 
performance-based  models  (frequency  and  sequence)  probably 
due  to  the  intrusiveness  of  the  dialogue.  The  lack  of  a 
user  preference  difference  for  the  plan-based  models  may 
argue  for  the  viability  of  computer-initiated  dialogues  with 
planning  information  since  verbal  explanations  may  alleviate 
some  of  the  psychological  intrusiveness  of  the  dialogue. 

Thus,  not  only  did  this  experiment  demonstrate  the 
effectiveness  of  the  command-selection  aids,  the  experiment 
also  provided  detailed  recommendations  for  the 
implementation  of  future  command-selection  aids. 
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Command- selection  models  and  associated  dialogue-initiatives 
worthy  of  future  research  and  development  in  this 
information  retrieval  task  are: 

1.  plan-based  models 

a)  user-initiated  dialogues 

b)  computer-initiated  dialogues 

2.  frequency  models 

a)  user-initiated  dialogues 

However,  these  recommendations  should  not  discourage  further 
investigation  of  other  models  or  dialogues  since  the 
knowledge  and  understanding  of  advisory  and  online  training 
systems  is  just  beginning  to  form. 
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Appendix  A 

INFORMED  CONSENT  FOR  RESEARCH  PARTICIPANTS 


A  LAN DX .RANT  UNIVERSITY 

VIRGINIA  POLYTECHNIC  INSTITUTE  AND  STATE  UNIVERSITY 

Human  Factors  Laboratory  BLckibur%  24061 

130  Whittemore  Hall 


PARTICIPANT'S  STATEMENT  OF  INFORMED  CONSENT 

>  ou  are  asked  to  participate  in  a  study  of  to  investigate  how  people  retrieve 
computer-based  information.  The  purpose  of  the  study  >s  to  provde  us  information 
on  how  to  assist  and  instruct  inexperienced  computer  users.  In  the  experiment  >  ou 
will  be  asked  to  retrieve  specific  information  from  a  hypothetical  database.  The  study 
will  take  approximately  10  total  hours  spread  across  six  sessions  on  separate  days. 
During  some  of  these  sessions  you  may  receive  computer-based  assistance  in  retrieving 
information  from  the  database. 

We  hope  that  this  experiment  will  be  an  interesting  experience  for  you.  It  is  possible 
that  at  times  you  may  feel  frustrated  or  stressed.  However,  your  performance  only 
reflects  the  difficulty  of  the  task. 

All  information  collected  in  the  experiment  will  be  held  in  strict  confidence.  We  will 
use  the  information  for  statistical  and  summary  purposes  only,  and  will  make  certain 
that  your  name  is  not  associated  with  your  reco-ds.  To  the  best  of  our  knowledge, 
there  a r«  no  physical  or  psychological  risks  associated  with  the  procedures  in  our 
study . 

As  a  participant  in  this  study,  you  have  certain  rights.  These  rights  will  now  be 
explained  to  you,  and  you  will  be  asked  for  your  signature,  indicating  that  you 
consent  to  participation  in  this  research. 

1.  You  have  the  right  to  stop  the  experiment  in  which  you  are  participating  at 
any  time  if  you  feel  that  it  is  not  agreeable  to  you.  Should  you  terminate  whe 
experiment,  you  will  receive  pay  only  for  the  proportion  of  time  you 
participated . 

2.  You  have  the  right  to  see  ycur  data  and  to  withdraw  it  from  the  experiment  if 
you  feel  that  you  should. 

3.  You  have  the  right  to  be  informed  of  the  results  of  the  overall  fexpenment.  If 
you  wish  to  receive  a  summary  of  the  results,  please  indicate  your  address 
(three  montns  hence)  with  your  signature.  A  summary  will  be  sent  to  you.  If 
you  should  then  like  further  information,  please  contact  the  Human  Factors 
Laboratory  and  a  full  report  will  be  made  available  to  you. 

4.  You  have  the  rignt  to  cal!  either  Dr.  Robert  Williges,  the  principal 
investigator,  at  961-6270  or  Mr.  Charles  Waring,  Institutional  Review  Board 
Chairman,  at  961-5233,  with  your  concerns  about  any  aspect  of  the  experiment. 

The  faculty  and  graduate  students  involved  greatly  appreciate  your  help  as  a 
participant.  If  you  have  any  question  about  the  experiment  or  your  rights  as  a 
participant,  please  do  not  hesitate  to  ask.  We  will  do  our  best  to  answer  them, 
subject  only  to  the  constraint  that  we  do  not  want  to  pre-bias  the  experimental 
results . 

Your  signature  on  the  back  of  this  form  indicates  that  you  have  read  your  rights  as  a 
participant  as  stated  above  and  that  you  consent  to  participation.  If  you  include  your 
printed  name  and  address  below,  a  summary  of  the  experimental  results  will  be  sent  to 
you  . 


181 


Appendix  B 

PORTIONS  OF  THE  INFORMATION  RETRIEVAL  DATABASE 


FILE  NUMBER  1 


Any  Operations 


Tank  Division 

Tank  Equipment  Statistics  (FILE  NUMBER  2) 

Tank  Division  Mission 

To  conduct  offensive  and  defensive  operations  on  a  battlefield 
either  independently  or  as  part  of  a  combined  arms  army. 

Tank  Division  Subgroups 

Division  Headquarters 
Tank  Regiments 
Motorized  Rifle  Regiments 

Tank  Division  Combat  Support  (FILE  NUMBER  B) 

Tank  Division  Operational  Capabilities 

Offense 

Attack  as  a  first  echelon  division  of  a  tank  army  to  penetrate 
enemy  defensive  positions 

Operate  as  the  exploitation  force  of  a  combined  arms  or  tank  army 
to  exploit  gaps  in  enemy  formations 
Attack  as  a  first  echelon  division  of  a  combined  army  whenever  the 
enemy  is  weak  and  good  terrain  is  available 
Conduct  pursuit  operations,  whenever  the  enemy  is  withdrawing,  to 
destroy  the  enemy  and  secure  army  objectives 

Defense 

Conduct  mobile  counterattack  operations  from  positions  within  or 
behind  the  army's  second  echelon 


Defend  in  place  to  gain  time  to  mass  forces 

Tank  Division  Personnel  Strengths 

for  continuing  an  attack 

Enlisted 

Unit 

Officers 

Personnel 

Division  Headquarters 

93 

208 

Tank  Regiments 

120 

981 

Artillery  Regiments 

98 

1033 

Mult.  Rocket  Launcher  Battalions 

30 

491 

Anti-Aircraft  Gun  Regiments 

26 

302 

Reconnaissance  Battalions 

39 

261 
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FILE  NUMBER  2 

Tank  Equipment  Statistics 
T-72  Battle  Tank 

T-72  Tank  Inventory  and  Availability  (FILE  NUMBER  3) 

Specific  Characteristics 

Crew i  Three  (commander*  gunner*  driver) 

Height i  40  tons 

Size 

Lengthi  9.02m  (gun  forwards) 

Hidthi  3.375m 
Height i  2.265m 

Road  speed i  80  km/hr 
Road  range i  500km 

Main  armament i  125mm  smooth  bore  gun 

Secondary  armament i  7.62mm  PICT  machine  gun  co-axial  with  the 
main  armament  and  a  12. 7mm  DshK  anti-aircraft  machine  gun. 

• 

Method  of  ranging i  Laser  rangefinder  and  an  infra-red  searchlight. 

Ammunition!  40  rounds  of  125mm  ammunition*  22  HE*  6  HEAT*  and  12 
APF5DS  rounds. 

Miscellaneous  Characteristics 

The  T-72  has  large  road  wheels  and  can  be  fitted  with  lightly 
armored  skirting  plates  to  provide  a  measure  of  protection 
against  HEAT  projectiles.  The  T-72  can  ford  to  a  depth  of  1.4m 
without  preparation  and  when  fitted  with  a  snorkel  can  ford 
to  5.486m.  The  engine  is  a  water  cooled  diesel  developing  700hp. 
This  provides  the  T-72  with  the  agility  to  climb  gradients  of 
up  to  60/..  It  can  also  breach  a  2.8m  trench  or  climb  a 
0.81m  vertical  obstacle. 

T-64  Battle  Tank 

T-64  Tank  Inventory  and  Availability  (FILE  NUMBER  4) 

Specific  Characteristics 

Crewi  Three  (commander*  gunner*  driver) 

Height i  35  tons 

Size 

Lengthi  8.0m 
Hidthi  3.375m 
Height i  2.25m 


Road  speed i  60  km /hr 
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FILE  NUMBER  3 

T-72  Tank  Inventory  and  Availability 
Number  of  T-72  Tanks 


Tank 

Regiment 

Total 

Total 

Regiment 

Name 

Number 

Available 

97 

Bravo 

20 

15 

356 

Lightning 

25 

20 

567 

Grayhawk 

19 

17 

789 

Spearhead 

17 

7 

890 

Arrow 

19 

18 

1679 

Wildcat 

15 

13 

Tank  Availability  and  Maintenance 
Bravo  Tank  Regiment  97 

Tank  Front  Maintenance  Battle  Maintenance 

ID  Location  Battalion  Hours  Code 

Yellow-1  25  Engine 

Yellow-1  33  Track 

Yellow- 1  45 

Orange -1  4 

Yellow-1  14 

Yellow-1  13  Turret 

Orange-1  10 

Orange-1  6 

Yellow-1  15  Gun 

Orange-1  56 

Yellow-1  13 

Orange-1  14 

Yellow-1  54 

Yellow-1  43 

Orange-1  18 

Orange- 1  7  Track 

Orange-1  12 

Orange-1  17 

Yellow-1  13 

Orange-1  4 

Lightning  Tank  Regiment  356 


Tank 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

4358 

Northwest 

Orange-2 

27 

2010 

Northwest 

Orange-2 

40 

64P2 

Northwest 

Yellow-2 

11 

9304 

Northwest 

Orange-2 

16 

Loader 

7301 

Northwest 

Orange-2 

19 

5617 

Northwest 

Orange-2 

18 

Turret 

9078 

North 

Orange-2 

15 

4977  North 
34 AA  North 
SAKE  North 
13E5  North 
49 ID  North 
2EC8  Northeast 
44 3F  North 
5122  North 
49D7  Northeast 
3YAA  Northeast 
5A4E  Northeast 
9265  Northeast 
4G16  Northeast 
2YC3  Northeast 
345F  Northeast 
5T2C  Northeast 
49D7  Northeast 
51 AT  North 
2A6E  North 
1295  North 
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FILE  NUMBER  4 

T-64  Tank  Inventory  and  Inventory 
Number  of  T-64  Tanks 


Tank 

Regiment 

Total 

Total 

Regiment 

Name 

Number 

Available 

45 

Charlie 

15 

10 

109 

Tower 

10 

5 

345 

Bulldog 

18 

9 

629 

Bullet 

13 

8 

878 

Razor 

18 

15 

1399 

Renegade 

14 

7 

Tank  Availability  and  Maintenance 


Tank  Regiment  45 

Tank 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

3897 

South 

Yellow-5 

15 

140  A 

South 

Yellow-5 

49 

5R79 

South 

Orange-5 

40 

13W5 

South 

Orange-5 

1 

4916 

South 

Yellow-5 

19 

2PC8 

South 

Yellow-5 

13 

Turret 

44 3F 

Southeast 

Orange-5 

35 

Track 

5569 

Southeast 

Orange-5 

9 

Gun 

43D0 

Southeast 

Orange-5 

22 

3Y2A 

Southeast 

Orange-5 

36 

5A4I 

On  reserve 

Ye  How -5 

10 

T2H5 

On  reserve 

Yellow-5 

19 

Armor 

441P 

On  reserve 

Yellow-5 

55 

2YI5 

South 

Yellow-5 

44 

Engine 

34ZQ 

South 

Orange-5 

19 

Tower  Tank  Regiment  109 


Tank 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

D258 

West 

Orange-7 

89 

2190 

Hest 

Orange-7 

10 

Loader 

24PP 

West 

Yellow-7 

71 

9914 

Hest 

Orange-7 

91 

Loader 

7321 

We3t 

Orange-7 

88 

56  3F 

Hest 

Orange-7 

45 

Turret 

9D71 

Southwest 

Yellow-7 

1 

Engine 

0235 

Southwest 

Yellow-7 

10 

Track 

48 2K 

Southwest 

Orange-7 

35 

7C1R 

Southwest 

Orange-7 

3 

Bulldog  Tank  Regiment  345 
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FILE  HUMBER  5 


T-62  Tank  Inventory  i 

and  Availability 

Number  of  T-62  Tanks 

Tank 

Regiment  Total 

Total 

Regiment 

Name 

Number 

Available 

103 

Laser 

12 

6 

221 

Alpha 

9 

9 

302 

Desert  10 

7 

721 

Scorpion  25 

10 

Tank  Availability 

and  Maintenance 

Laser  Tank  Regiment  103 

Tank 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

X082 

Northeast 

Orange-1 

5 

Turret 

131A 

Northeast 

Yellow-1 

39 

5R69 

Northeast 

Orange-1 

20 

Loader 

33W8 

North 

Orange-1 

5 

4946 

North 

Yellow-1 

19 

2P12 

North 

Yellow-1 

15 

483F 

North 

Yellow-1 

37 

Track 

5ZH9 

Northeast 

Orange-1 

10 

Gun 

4000 

Northeast 

Yellow-1 

21 

3YSA 

Northeast 

Orange- 1 

37 

Gun 

5A41 

Northeast 

Yellow-1 

19 

6ZH3 

Northeast 

Yellow-1 

13 

Armor 

Alpha  Tank  Regiment  221 

Tank 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

D258 

South 

Orange-6 

30 

2190 

South 

Orange-6 

20 

24PP 

South 

Yellow-6 

62 

9914 

South 

Orange-6 

63 

7321 

South 

Yellow-6 

25 

56  3F 

West 

Orange-6 

24 

9071 

Hest 

Yellow-6 

5 

0235 

West 

Orange-6 

15 

482K 

West 

Orange-6 

22 

Desert  Tank  Regiment  302 

Tank 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

IDYL 

South 

Yellow-5 

45 

2BT1 

South 

Yellcw-5 

72 

Track 
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FILE  NUMBER  6 

T-10  Tank  Inventory  and  Availability 
Number  of  T-10  Tanks 


Tank 

Regiment 

Total 

Total 

Regiment 

Name 

Number 

Available 

23 

Hammer 

10 

5 

723 

Bear 

10 

7 

1826 

Victory 

5 

4 

Tank  Availability  and  Maintenance 


Hammer  Tank  Regiment  23 


Tank 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

13Y1 

On  reserve 

Yellow-5 

10 

Turret 

34 9 A 

On  reserve 

Orange-5 

28 

Track 

7369 

On  reserve 

Orange-5 

27 

Turret 

33U8 

On  reserve 

Orange-5 

10 

5946 

On  reserve 

Yellow-5 

1 

Loader 

2342 

Southeast 

Yellow-5 

10 

4YSF 

Southeast 

Yellow-5 

23 

8MN9 

Southeast 

Yellow-5 

39 

Gun 

10DW 

Southeast 

Yellow-5 

20 

3T5A 

Southeast 

Orange-5 

33 

Bear  Tank  Regiment  723 

Tank 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

821L 

South 

Orange-8 

33 

2721 

South 

Orange-B 

43 

Loader 

1357 

Southeast 

Yellow-8 

81 

5KP3 

Southeast 

Orange-8 

24 

Z1F6 

Southeast 

Orange-8 

1 

2SC7 

Southeast 

Yellow-8 

42 

34  3X 

On  reserve 

Ye  How -8 

45 

1490 

On  reserve 

Yellow-8 

23 

Track 

5440 

On  reserve 

Orange-8 

21 

Turret 

5544 

On  reserve 

Yellow -8 

22 

Victory  Tank  Regiment  1826 


Tank 

Front 

Maintenance 

Battle 

ID 

Location 

Battalion 

Hours 

1T26 

North 

Orange-4 

37 

48P0 

North 

Yellow-4 

38 

2EL2 

North 

Orange-4 

32 

8E23 

Hortheast 

Orange-4 

17 

Maintenance 

Code 


Turret 
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FILE  NUMBER  7 


PT-76  Tank  Inventory  and  Availability 
Number  of  P7-76  Tanks 


Tank 

Regiment 

Total 

Total 

Regiment 

Name 

Number 

Avails 

56 

Aqua 

25 

18 

178 

Neptune 

22 

20 

358 

Delta 

30 

25 

985 

River 

20 

10 

1093 

Seal 

15 

9 

Tank  Availability  and  Maintenance 
Aqua  Tank  Regiment  56 


Tank 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

155U 

South 

Green-S 

15 

8922 

South 

Green-3 

25 

Turret 

1G90 

South 

Blue-3 

23 

5E89 

South 

Green-3 

43 

Loader 

58  Y  2 

South 

Blue-3 

1 

783P 

On  reserve 

Green-3 

14 

Turret 

3902 

On  reserve 

Blue-3 

9 

4828 

On  reserve 

Blue-3 

20 

39S3 

On  reserve 

Green-3 

15 

4W0I 

On  reserve 

Blue-3 

16 

3 94 A 

On  reserve 

Green-3 

53 

5931 

Southwest 

Blue-3 

62 

1T49 

Southwest 

Blue-3 

29 

2032 

Southwest 

Green-3 

4 

Turret 

2902 

Southwest 

Blue-3 

56 

0P98 

Southwest 

Green-3 

37 

9458 

Southwest 

Green-3 

34 

2Q03 

Southwest 

Green-3 

67 

Haterjet 

19K0 

North 

Blue-1 

53 

490F 

North 

Blue-1 

34 

5902 

Northeast 

Blue-1 

27 

Gun 

43F5 

Northeast 

Green-1 

38 

9345 

Northeast 

Green-1 

37 

Gun 

2590 

Northeast 

Green-1 

21 

2150 

Northeast 

Green-1 

18 

Neptune  Tank  Regiment  178 


Tank 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

92N8 

Southeast 

Green-2 

31 

29S3 

Southeast 

Green-2 

25 

8043 

Southeast 

Blue-2 

36 
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FILE  NUMBER  8 


Tank  Division  Combat  Support 
Artillery  Regiment 

Artillery  Regiment  Mission 

Provide  fire  support  to  the  tank  divisions  making  a  main  advance 

Artillery  Regiment  Subgroups 

Headquarters 
Artillery  Units 
Anti-Tank  Battalions 
Motor  Transport  Battalions 
Target  Acquisition  Units 

Artillery  Regiment  Operational  Capabilities 

Deliver  preparatory  fires  to  support  the  offensive 
Engage  anti-tank  defenses 
Deliver  fire  on  infantry  strong  points 
Deliver  smoke  cover  on  enemy  observation  points 
Deliver  illumination  and  marking  rounds  in  night  operations 
for  friendly  directional  control 
Deliver  harassing  and  interdicting  fire  on  enemy  routes  of 
reinforcement  and  withdrawal 

Multiple  Rocket  Launcher  Battalion 

Multiple  Rocket  Launcher  Battalion  Mission 

To  provide  a  multi-barreled  rocket  launcher  capability  in  support 
of  division  operations 

Multiple  Rocket  Launcher  Battalion  Subgroups 

Headquarters 
Firing  Battery 
Service  Battery 

Multiple  Rocket  Launcher  Battalion  Operational  Capabilities 

Provide  heavy  weight  of  fire  on  important  targets  at  decisive 
moments  in  an  engagement 
Provide  a  rapid  fire  capability 
Provide  counter-bombardment 

Anti-Aircraft  Gun  Regiment 

Anti-Aircraft  Gun  Regiment  Mission 

Provide  close-in  air  defense  protection  to  the  division  by 
engaging  targets  at  maximum  operational  ranges  during 
offensive  and  defensive  operations 
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FILE  HUMBER  9 


Motorized  Rifle  Equipment  Statistics 
BMP-1  Armored  Personnel  Carrier 

BMP-1  Inventory  and  Availability  I  FILE  HUMBER  10) 

Specific  Characteristics 

Crew i  11  (Commander*  gunner*  driver  plus  8  infantrymen) 
Height*  12.5  tons 

Size 

Length*  6.75m 
Width*  3.0m 
Height*  2.0m 

Road  speed*  55  km/hr 
Road  range*  500km 

Main  armament*  73mm  low  pressure  gun 

Secondary  armament*  7.62mm  PKT  machine  gun  co-axial  with  the 
main  armament 

Method  of  ranging*  Visual  and  infra-red  searchlight 

Ammunition*  28  HEAT  rounds 

Miscellaneous  Characteristics 

The  BMP-1  is  designed  to  accompany  tanks  into  the  assault 
and  right  on  to  and  beyond  the  objective.  Although  the 
the  BMP-1  is  an  amphibian*  it  does  not  employ  hydrojets*  but 
obtains  its  propulsion  in  water  from  the  the  tracks.  Exits 
for  the  infantry  section  is  through  two  rear  doors*  which 
have  fuel  tanks  incorporated.  The  combination  of  effective 
anti-armor  fire  power*  high  mobility*  and  adequate  protection 
makes  the  BMP-1  an  formidable  enemy. 

BTR-60  Armored  Personnel  Carrier 

BTR-60  Inventory  and  Availability  (FILE  HUMBER  11) 

Specific  Characteristics 

Crew*  12  (Driver*  commander  plus  10  infantrymen) 

Height*  10  tons 

Size 

Length*  7.56m 
Width*  2.82m 
Height*  2.31m 

Road  speed*  80  km/hr 
Road  range*  500km 
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FILE  NUMBER  10 


BMP-1  Armored  Personnel  Carrier  (APC-  Inventory 
Number  of  BMP-1  A PCs 


APC 

Regiment 

Total 

Total 

Regiment 

Name 

Number 

Avail; 

25 

Sahara 

40 

32 

278 

Prairie 

53 

47 

892 

Harrow 

85 

56 

931 

Zulu 

45 

21 

APC  Availability  and  Maintenance 


Sahara  APC  Regiment  25 


APC 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

4B675 

Northwest 

Orange -2 

78 

Track 

884 7C 

Northwest 

Orange-2 

85 

BBB63 

Northwest 

Yellow-2 

65 

CAS  19 

Northwest 

Orange-2 

24 

Fuel 

918D4 

Northwest 

Yellow-2 

124 

DC623 

Northwest 

Yellow-2 

10 

854A8 

Northwest 

Yellow-Z 

170 

8E477 

North 

Yellow-2 

147 

D61BF 

North 

Orange-2 

13 

BC1DA 

North 

Yellow-2 

75 

Track 

6CBA1 

North 

Yellow-Z 

35 

99F14 

North 

Orange-Z 

75 

327E7 

North 

Yellow-2 

101 

98137 

North 

Yellow-2 

157 

Hydrojet 

B5C15 

North 

Yellow-2 

199 

6F395 

North 

Yellow-2 

35 

E1B3 

North 

Yellow-2 

2 

5E0C3 

North 

Yellow-2 

54 

ABCBF 

West 

Green-2 

140 

Fuel 

1101F 

West 

Green -2 

11 

B6DA2 

West 

Green-2 

105 

ECE5E 

North 

Yellow-2 

47 

C396 

North 

Yellow-Z 

195 

Track 

DC  944 

North 

Yello  -2  _ 

20 

1CCAC 

North 

Orange-2 

115 

8BZ69 

North 

Orange-2 

2 

6B609 

North 

Orange-2 

151 

80FA3 

North 

Orange-2 

161 

598DS 

North 

Orange-2 

191 

Gun 

5AD68 

West 

Green-2 

125 

Gun 

DD4E9 

West 

Green-2 

57 

6E8D0 

West 

Green-2 

132 

69024 

West 

Green-2 

11 

A08ZE 

West 

Green-2 

64 

3223E 

West 

Yellow-2 

41 

FILE  NUMBER  11 


BTR-60  Armored  Personnel  Carrier  (APC)  Inventory 
Number  of  BTR-60  A  PCs 


APC 

Regiment  Total 

Total 

Regiment 

Name 

Number 

Avail: 

102 

Forest  34 

30 

472 

Thunder  46 

75 

499 

Tiger 

39 

35 

505 

Echo 

40 

26 

634 

Sierra  53 

45 

Lability 

and  Maintenance 

:  APC  Regiment  102 

A  PC 

Front 

Maintenance 

Battle 

ID 

Loeat  ion 

Battalion 

Hours 

7D4C2 

On  reserve 

Orange-1 

19 

272AE 

On  reserve 

Otarge-1 

44 

1CF92 

On  reserve 

Orange-1 

95 

B9A7E 

On  reserve 

Orange-1 

149 

4AAE3 

On  reserve 

Yellow-4 

88 

30FA0 

Northeast 

Yellcw-4 

125 

A7EB6 

Northeast 

Yellow-4 

59 

117DF 

Northeast 

Yellow-4 

78 

E4A32 

North 

Yellow-4 

59 

2F57 

North 

31ue~4 

82 

3BE7E 

North 

Blue -4 

134 

A006D 

North 

Blue-4 

162 

Su550 

North 

Blue-1 

133 

AEF2D 

North 

Blue-1 

89 

9C067 

North 

Blue-1 

13 

1E7A 

Northeast 

Yellow-1 

12 

CF05D 

Northeast 

Yellow-1 

125 

9DA10 

Northeast 

Yellow-1 

50 

22AB 

Northeast 

Yellow-1 

41 

662C9 

Northeast 

Yellow-1 

154 

8B526 

Northeast 

Yellow-1 

76 

B4EF7 

Southeast 

Yellow-1 

100 

C49CA 

Sor ' heast 

Yellow-1 

10 

5C21F 

North 

Yellow-1 

144 

D6C06 

Southeast 

Yellow-1 

45 

9D925 

Sou  bheast 

Yellow-4 

95 

AB82C 

Southeast 

Yellow-4 

30 

39E6B 

Southeast 

Orange-1 

161 

7B768 

Southeast 

Lrange-l 

41 

8EBE3 

Southeast 

Orange-1 

77 

D3048 

On  reserve 

Blue-I 

11 

958C2 

On  reserve 

Blue-1 

153 

50DF5 

On  reserve 

Blue-1 

149 

9539A 

On  reserve 

Blue-1 

126 

Maintenance 

Code 


Hydrojet 
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FILE  NUMBER  12 


[-50P  Armored  Personnel 

Carrier  (APC) 

Inventory 

Number  of  BTR-SOP  APCs 

APC 

Regiment 

Total 

Total 

Regiment 

Name 

Number 

Available 

12 

Caspian 

46 

37 

24S 

Oscar 

49 

45 

452 

Titan 

45 

531 

Marauder 

37 

634 

Silent 

50 

49 

1983 

Crater 

59 

59 

APC  Availability  and  Maintenance 

Caspian  A PC  Regiment  12 


APC 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

5931 

East 

Orange -3 

282 

E0113 

Fast 

Orange-3 

109 

Cooling 

8C841 

East 

Orange-3 

192 

74C5B 

East 

Orange-8 

190 

23CB9 

East 

Orange-8 

170 

9943 

East 

Orange-8 

70 

D2C16 

East 

Orange-8 

97 

Engine 

D7143 

On  reserve 

Orange-8 

237 

Engine 

9C65E 

On  reserve 

Orange-8 

63 

4B6F7 

On  reserve 

Blue-5 

217 

72574 

On  reserve 

Blue-5 

71 

4CB65 

On  reserve 

Blue -5 

153 

34037 

On  reserve 

Blue -5 

205 

CAD  SB 

Southeast 

Blue-5 

311 

6715A 

Southeast 

Orange-8 

317 

Gun 

89253 

Southeast 

Orange-8 

307 

Fuel 

EAB17 

Southeast 

Orange-8 

241 

32B21 

So'  theast 

Orange -3 

150 

6E5C2 

Southeast 

Orange-3 

48 

BCED9 

Southeast 

Orange-3 

161 

4B124 

Southeast 

Orange-3 

179 

A9BDE 

Southeast 

Orange-8 

109 

892EC 

On  reserve 

Orange-8 

261 

Fuel 

19032 

On  reserve 

Orange-8 

170 

'  93 

On  reserve 

Orange-8 

268 

C3BFC 

On  reserve 

Orange-8 

260 

99752 

West 

Orange-8 

169 

4F'C8 

West 

Orange-8 

306 

9CFOO 

West 

Orange-8 

314 

Z479A 

West 

Orange-8 

116 

Hydro jet 

47C93 

West 

Orange-8 

255 

EF0D7 

West 

Orange-  3 

152 

DRC64 

West 

Orange-3 

305 
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FILE  NUMBER  13 


BTR-152  Armored  Personnel  Carrier 

(APC)  Inventory 

Number  of  BTR-152 

A  PCs 

APC 

Regiment  Total 

Total 

Reg iment 

Name 

Number 

Available 

143 

Ranger  20 

9 

-  780 

Lion 

25 

14 

854 

Taurus  15 

11 

929 

Leopard  10 

5 

APC  Availability  ; 

and  Maintenance 

Ranger  APC  Regiment  143 

APC 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

6784 

South 

Yellow-7 

42 

67244 

South 

Yellow-7 

7 

3A219 

South 

Yellow-7 

13 

Armor 

85FCD 

South 

Yellow-7 

15 

Fuel 

1873F 

South 

Yellow-7 

42 

215AB 

Southeast 

Yellow-7 

20 

Wheel 

B5B8E 

Southeast 

Orange-2 

9 

B2208 

East 

Yellow-7 

47 

AFE42 

East 

Orange-2 

38 

9767C 

East 

Orange-2 

50 

68A62 

East 

Yellow-7 

67 

18682 

On  reserve 

Orange-2 

44 

Engine 

EA4A6 

On  reserve 

Orange-2 

41 

Engine 

CA01F 

On  reserve 

Orange-2 

57 

Armor 

DC888 

On  reserve 

Orange-2 

14 

Engine 

06070 

On  reserve 

Yellow-7 

3 

Fuel 

D742F 

On  reserve 

Orange-2 

37 

Wheel 

BF8B3 

On  reserve 

Orange-2 

48 

D576E 

On  reserve 

Orange-2 

59 

6E692 

On  reserve 

Orange-2 

21 

Wheel 

Lion  APC  Regiment  780 

APC 

Front 

Maintenance 

Battle 

Maintenance 

ID 

Location 

Battalion 

Hours 

Code 

E3CFC 

Southeast 

Blue-1 

63 

ECB33 

Southeast 

Blue-1 

64 

Armor 

43400 

Southeast 

Yellow-7 

9 

4  AC  94 

Southeast 

Blue-1 

60 

80D8A 

Southeast 

Blue-1 

69 

9EF13 

On  reserve 

Blue-1 

64 

Engine 

CA2F8 

On  reserve 

Blue-1 

22 

Engine 

71C86  On  reserve  Orange-4  54 

835E1  On  reserve  Orange-4  34 


Engine 
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FILE  NUMBER  14 


MT-LB  Multipurpose  Tracked  Vehicle  (MTV )  Inventory 
Number  of  MT-LB  MTV 3 


MTV 

Regiment 

Total 

Total 

Regiment 

Name 

Number 

Available 

108 

Bison 

49 

46 

361 

Gray 

53 

50 

571 

Rover 

45 

43 

780 

Cheetah 

42 

41 

853 

Foxtrot 

45 

40 

1276 

Whiskey 

50 

45 

1852 

Xenon 

52 

49 

1871 

Pelican 

46 

40 

MTV  Availability  and  Maintenance 
Bison  MTV  Regiment  108 


MTV 

Front 

Maintenance 

Battle 

ID 

Location 

Battalion 

Hours 

845EF5 

East 

Green -2 

164 

73FD59 

East 

Green-2 

308 

50BF32 

East 

Green-2 

114 

8CDA2B 

East 

Green-2 

281 

22C7AF 

East 

Green-2 

198 

4FB04B 

East 

Green-2 

102 

22FF50 

East 

Green-2 

162 

6ACED8 

East 

Green-2 

313 

6392B5 

East 

Green-2 

133 

196054 

East 

Green-2 

118 

51077A 

East 

Green-2 

103 

80205 

East 

Green-2 

68 

EDE7F 

East 

Green-2 

297 

1F5AD4 

East 

Green-2 

237 

862FF5 

East 

Green-2 

7 

75FDE5 

East 

Green-2 

53 

1F398E 

East 

Green-2 

350 

3F1F7A 

East 

Green-2 

55 

551D46 

East 

Green-2 

63 

396C31 

East 

Green-2 

384 

OHF28 

East 

Green-2 

331 

tr.85F2 

East 

Green-2 

226 

->/AD67 

Southeast 

Green-2 

392 

45C3BE 

Southeast 

Green-2 

3 

51DE0C 

Southeast 

Green-2 

318 

8D1715 

Southeast 

Green-2 

179 

6A9958 

Southeast 

Green-2 

102 

390AF4 

Southeast 

Green-2 

250 

9BF50 

Southeast 

Green-2 

132 

938B6B 

Southeast 

Green-2 

153 

269598 

Southeast 

Green-2 

233 

Maintenance 

Code 


Armor 


Gun 


Engine 
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FILE  HUMBER  15 


Motorized  Rifle  Division  Combat  Support 
Direct  Combat  Support 
Artillery  Regiment 

Artillery  Regiment  Mission 

Provide  fire  support  to  the  notorized  rifle  divisions  making 
a  main  advance 

Assist  in  the  defense  of  critical  areas 

Artillery  Regiment  Subgroups 

Headquarters 
Artillery  Units 
Target  Acquisition  Units 

Artillery  Regiment  Operational  Capabilities 

Deliver  preparatory  fires  to  support  offensive 
Suppress  enemy  artillery 
Deliver  fire  on  infantry  strong  points 
Deliver  fire  on  -enemy  tactical  reserves 
Deliver  smoke  cover  on  enemy  observation  points 
Deliver  illumination  and  marking  rounds  in  night  operations 
for  friendly  directional  control 
Deliver  harassing  and  interdicting  fire  on  enemy  routes  of 
reinforcement  and  withdrawal 
Engage  targets  of  opportunity 

Destroy  major  supply  depots  and  communications  in  rear  areas 

Anti-Tank  Missile  Battalion 

Anti-Tank  Missile  Battalion  Mission 

To  provide  anti-tank  support  to  motorized  rifle  forces  as 
the  nucleus  of  the  anti-tank  reserve 

Anti-Tank  Missile  Battalion  Subgroups 

Headquarters 

Anti-Tank  Missile  Battery 
Supply  and  Maintenance  Company 

Anti-Tank  Missile  Battalion  Operational  Capabilities 

Offense 

Move  behind  the  advancing  first  echelon  tanks  and  infantry 
on  the  most  exposed  line  of  advance*  ready  to  cover  the 
deployment  of  enemy  attacking  units  and  to  repel  areor 
counterattacks 
Cover  the  flanks 


Appendix  C 

RECOGNITION  TEST  FOR  THE  DATABASE 


1.  Which  of  the  following  is  a  subfile  of  Tank  Equipment 
Statistics? 

a.  BTR-60  Inventory  and  Availability 

b.  Artillery  Equipment  Statistics 

c.  PT-76  Inventory  and  Availability 

d.  Tank  Division  Combat  Support 

e.  F-139  Inventory  and  Availability 

z.  What  file  would  a  maintenance  battalion,  maintenance 
code  and  number  of  battle  hours  be  listed  in? 

a.  Armored  Vehicle  Equipment  Statistics 

b.  Tank  Equipment  Statistics 

c.  BMP-1  Inventory  and  Availability 

d.  Army  Operations 

e.  Tank  Division  Combat  Support 

3.  Where  besides  the  Army  Operations  file  will  operational 
capabilities  be  found? 

a.  Tank  Division  Combat  Support 

b.  Tank  Equipment  Statistics 

c.  FT-21  Inventory  and  Availability 

d.  Anti-Aircraft  Inventory  and  Availability 

e.  PT-76  Inventory  and  Availability 

4.  Which  of  the  following  files  is  one  file  below  the  Army 
Operations  in  the  file  hierarchy? 

a.  Multiple  Rocket  Launcher  Equipment 

b.  SBT-132  Inventory  and  Availability 

c.  Tank  Division  Combat  Support 

d.  PT-76  Inventory  and  Availability 

e.  T-62  Inventory  and  Availability 
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5.  Information  on  crew  size  is  contained  in: 

a.  Motorized  Rifle  Equipment  Statistics 

b.  Tank  Division  Combat  Support 

c.  BTR-50P  Inventory  and  Availability 

d.  Army  Operations 

e.  Surface  to  Air  Missile  Inventory 

6.  What  file  has  information  on  armored  personnel  carriers? 

a.  Motorized  Rifle  Division  Combat  Support 

b.  Motorized  Rifle  Equipment  Statistics 

c.  Tank  Equipment  Statistics 

d.  Armored  Vehicle  Equipment  Statistics 

7.  Regiment  names  and  ID  codes  would  be  located  in: 

a.  BMP-1  Inventory  and  Availability 

b.  Armored  Vehicle  Equipment  Statistics 

c.  Army  Operations 

d.  Tank  Division  Combat  Support 

8.  In  which  of  the  following  files  is  front  information  is 
NOT  listed? 

a.  BMP-1  Inventory  and  Availability 

b.  Tank  Equipment  Statistics 

c.  T-62  Inventory  and  Availability 

d.  BTR-152  Inventory  and  Availability 

9.  A  code  like  Green-7  indicates  a: 

a.  Tank  regiment 

b.  Maintenance  battalion 

c.  Front  location 

d.  Division  subgroup 

e.  Ammunition  type 

10.  Information  on  the  total  number  of  officers  is  contained 
in  which  of  the  following  files? 

a.  Tank  Division  Combat  Support 

b.  Motorized  Rifle  Division  Combat  Support 

c.  Army  Operations 

d.  F- 10  Inventory  and  Availability 

e.  T-62  Inventory  and  Availability 
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11.  Battle  hours  is  included  in  which  of  the  following  files? 

a.  Artillery  Equipment  Statistics 

b.  Army  Operations 

c.  Tank  Division  Combat  Support 

d.  Motorized  Rifle  Division  Combat  Support 

e.  T-72  Inventory  and  Availability 

12.  Information  on  offensive  and  defensive  operational 
capabilities  is  NOT  contained  in: 

a.  T-10  Inventory  and  Availability 

b.  Army  Operations 

c.  Tank  Division  Combat  Support 

d.  Motorized  Rifle  Division  Combat  Support 

13.  Information  on  subgroups  and  mission  CANNOT  be  found  in: 

a.  Tank  Division  Combat  Support 

b.  Motorized  Rifle  Division  Combat  Support 

c.  Army  Operations 

d.  BTR-50P  Inventory  and  Availability 

14.  The  MT-LB  Inventory  and  Availability  file  would  be  a 
subfile  of: 

a.  Motorized  Rifle  Equipment  Statistics 

b.  Artillery  Equipment 

c.  Army  Operations 

d.  Armored  Vehicle  Equipment  Statistics 

e.  SBT-132  Inventory  and  Availability 


Appendix  D 

INTRODUCTORY  INSTRUCTIONS 


Welcome  to  the  Information  Retrieval  Assistance  Study. 

I  am  the  DECtalk,  and  you  will  be  listening  to  me  during 
the  training  phase  of  the  experiment. 

The  purpose  of  this  study  is  to  investigate  the  methods 
and  procedures  that  experienced  and  inexperienced  people  use 
in  information  retrieval  so  that  we  can  construct  systems 
that  assist  inexperienced  users  in  these  tasks. 

Therefore,  over  the  next  6  sessions  you  will  be  asked  to 
learn  and  use  an  information  retrieval  system  that  has  data 
on  a  hypothetical  enemy.  As  Jay  has  probably  already  told 
you,  today  is  probably  the  longest  session  since 
familiarization  and  training  on  the  procedures  of  the 
experiment  must  be  completed. 

This  training  session  will  probably  last  2  to  3  hours. 
The  next  5  sessions  will  probably  last  1  to  2  hours.  In 
these  last  5  sessions,  you  will  be  asked  to  locate 
information  in  an  information  retrieval  database.  In  all  of 
these  sessions,  you  will  be  given  breaks  to  avoid  any 
unnecessary  fatigue. 

The  tasks  and  procedures  that  we  are  asking  you  to 
perform  may  not  be  easy.  However,  we  hope  that  you  attempt 
to  perform  these  activities  as  best  you  can.  Throughout  the 
experiment  we  would  like  you  to  locate  the  information  as 
guickly  as  possible. 

Please  feel  fiee  to  ask  questions  during  the  experiment. 
This  is  especially  critical  today,  since  you  will  be  taught 
a  significant  amount  of  new  information  and  it  is  impossible 
to  explain  everything  clearly.  We  will  try  our  best  to 
answer  these  questions,  subject  only  to  the  constraint  that 
we  do  not  want  to  bias  the  results. 

Two  things  will  be  covered  today.  First,  you  will  be 
trained  and  familiarized  with  the  information  retrieval 
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database.  Second,  you  will  be  trained  and  practice  the 
search  procedures  individually  in  the  information  retrieval 
system. 

The  second  session  will  be  the  final  day  of  practice 
before  the  start  of  the  experiment.  In  this  session,  you 
will  first  review  the  database  and  search  procedures.  Then 
we  will  ask  you  to  locate  information  using  all  the  search 
procedures.  This  will  give  you  practice  integrating  the 
knowledge  and  skills  you  learned  on  the  first  day. 

At  the  end  of  this  second  session,  a  comprehension  test 
will  be  given  to  determine  your  knowledge  and  understanding 
of  the  database.  From  this  test,  and  a  measure  of  your 
speed  of  finding  information,  a  decision  will  be  made 
whether  you  can  continue  in  the  experiment. 

If  you  do  not  continue,  you  will  receive  payment  for  the 
proportion  of  time  you  participated. 

At  this  time,  please  ask  the  experimenter  to  start  you  on 
the  information  retrieval  database. 


Appendix  E 

TUTORIAL  FOR  THE  INFORMATION  RETRIEVAL  DATABASE 


Information  Retrieval  Database 

The  information  retrieval  database  is  a  large  set  of 
structured  files  that  contain  data  on  the  operations  and 
equipment  of  a  hypothetical  enemy.  If  the  military  nature 
of  this  database  offends  you,  we  would  like  to  apologize. 

However,  the  data  that  we  collect  from  this  study  is  not 
pointed  towards  any  specific  military  system.  Rather  the 
military  database  was  chosen  to  be  representative  of  the 
complexities  of  several  online  retrieval  systems.  In  other 
words,  our  results  are  focused  on  acquiring  basic  knowledge 
for  the  design  of  human-computer  systems,  in  general,  and 
information  retrieval  systems,  specifically. 

Two  pieces  of  information  to  your  left  will  be  used  in 
this  training  exercise.  First,  there  is  a  large  single 
piece  of  paper  which  contains  a  map  of  the  information 
retrieval  database.  Second,  there  is  a  3-ring. notebook 
which  contains  the  actual  database. 

Lets  take  a  look  at  the  map  of  the  information  retrieval 
database.  The  first  thing  you  should  know  about  this  map  is 
that  each  box  in  this  hierarchy  represents  a  file.  In  each 
box  is  a  label  for  the  information  that  is  contained  in  the 
file . 

Therefore,  the  information  retrieval  database  consists  of 
an  hierarchy  of  15  files.  These  files  are  divided  into  two 
major  divisions:  those  files  dealing  with  the  Tank  Division, 
and  those  files  dealing  with  the  Motorized  Rifle  Division. 

We  will  be  reviewing  each  of  the  15  files  briefly  to  give 
y-'U  a  familiarity  with  the  file  hierarchy.  Please  pay  close 
attention  to  this  review  since  your  continuation  in  this 
study  requires  that  you  understand  both  the  structure  and 
some  content  of  the  database. 
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In  fact,  as  was  explained  earlier  you  will  be  given  a 
comprehension  test  at  the  end  of  the  second  day  of  the 
experiment.  In  this  test,  we  will  ask  you  to  recognize 
important  aspects  of  the  database. 

However,  you  should  not  become  overly  concerned  with  this 
test  or  the  large  amount  of  information.  In  this  training 
session,  we  will  identify  the  important  information  that  you 
will  need  to  know.  The  rest  of  the  information  you  can  just 
scan  quickly. 

The  order  in  which  we  will  review  the  files  in  the  file 
hierarchy  will  be  by  the  red  numbers  listed  above  the  upper 
right  hand  corner  of  each  box  in  the  map.  This  number 
corresponds  to  the  file  number  in  the  tabs  of  the  notebook, 
and  the  file  number  in  the  upper  right  hand  corner  of  each 
page  in  the  notebook. 

Please  use  both  the  map  and  the  notebook  in  reviewing 
this  database. 


File  Number  1:  Army  Operations 

The  first  file  we  will  review  is  the  main  Army  Operations 
file.  This  is  the  top  file  of  the  hierarchy  and  will  serve 
as  the  starting  point  in  retrieval. 

A  point  to  note  about  this  file  and  all  other  files  is 
the  section  numbering  used  to  indicate  the  internal  file 
structure.  Each  file  has  has  an  internal  hierarchy  that 
will  be  used  by  the  information  retrieval  software  to 
present  a  table  of  contents. 

However,  the  section  numbers  will  not  be  displayed  in  the 
computerized  version  of  these  files.  Therefore,  it  is 
important  to  browse  these  lines  so  that  you  are  familiar 
with  the  internal  file  hierarchy. 

For  example,  the  important  sections  of  the  Army 
Operations  file  consist  of  information  on  the: 

1.  Tank  Division,  (Section  1.1) 

2.  Motorized  Rifle  Division  (Section  1.2). 

Each  of  these  sections  has  information  in  this  file  on: 

1.  equipment  statistics, 

2.  mission, 

3.  subgroups, 

4.  combat  support. 
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5.  operational  capabilities, 

6.  personnel  strengths. 

Please  browse  over  the  Army  Operations  file  and  note 
these  sections. 

In  addition,  with  file  number  1  you  should  also  note  the 
correspondence  between  the  highlighted  lines  and  the  map  of 
the  file  hierarchy.  Each  of  the  highlighted  lines  in  the 
file  corresponds  to  one  of  the  files  in  the  hierarchy.  You 
can  identify  the  match  between  the  highlighted  line  and  the 
file  in  the  hierarchy  by  the  section  or  file  number. 

The  section  number  of  the  highlighted  line  identifies 
where  this  line  is  found  in  the  current  file.  The  file 
number  in  the  "(TO  FILE  NUMBER  #)"  statement  identifies  the 
file  where  additional  information  can  be  found. 

So,  as  can  be  seen  through  the  map  and  the  highlighted 
lines  of  the  Army  Operations  file,  the  subfiles  for  this 
main  file  are: 

1.  Tank  Equipment  Statistics  (Section  1.1.1), 

2.  Tank  Division  Combat  Support  (Section  1.1.4), 

3.  Motorized  Rifle  Equipment  Statistics  (Section  1.2.1), 

4.  Motorized  Rifle  Division  Combat  Support 
( Section  1.2.4). 

In  addition,  the  file  numbers  to  the  right  of  each  of 
these  highlighted  lines  indicates  that: 

1.  Tank  Equipment  Statistics  can  be  found  in  file 
number  2 . 

2.  Tank  Division  Combat  Support  can  be  found  in  file 
number  8. 

3.  Motorized  Rifle  Equipment  Statistics  can  be  found 
in  file  number  9. 

4.  Motorized  Rifle  Division  Combat  Support  can  be  found 
in  file  number  15. 

Please  take  another  look  at  the  Army  Operations  file 
(number  1 ) . 


File  Number  2:  Tank  Equipment  Statistics 

This  file  includes  specific  and  miscellaneous  information 
on  the  following  tanks: 

1.  T-72  Battle  Tank  (Section  1.1. 1.1), 

2.  T-64  Battle  Tank  (Section  1.1. 1.2), 
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3.  T-62  Battle  Tank  (Section  1.1. 1.3), 

4.  T- 10  Heavy  Tank  (Section  1.1. 1.4), 

5.  PT-76  Amphibious  Tank  (Section  1.1. 1.5). 


On  each  of 
listed  are: 


these  tanks  the  specific  characteristics 


1.  Crew, 

2.  Weight, 

3.  Size, 

4.  Road  speed  and  range, 

5.  Armament, 

6.  Method  of  ranging, 

7.  Ammunition. 


Notice  that  this  file,  and  every  subsequent  file,  lists 
where  the  current  file  is  referenced.  In  this  case,  file 
number  2  was  referenced  by  file  number  1. 

In  addition,  notice  that  the  highlighted  lines  in  Tank 
Equipment  Statistics  (file  number  2)  access  subfiles 
associated  with  the  availability  and  inventories  of  each 
type  of  tank. 

That  is  to  say  the  T-72,  T-64,  T-62,  T-10,  and  PT-76  Tank 
Inventories  and  Availabilities  are  contained  in  file  numbers 
3,  4,  5,  6,  and  7,  respectively. 

Please  take  another  look  at  the  Tank  Equipment  Statistics 
file  ( number  2 ) . 

File  Numbers  3-7:  Tank  Inventory  and  Availability 

These  files  contain  information  about  the  availability 
and  inventories  of  each  tank  listed  in  Tank  Equipment 
Statistics  (file  number  2). 

At  the  top  of  each  of  these  files  is  a  section  containing 
information  about  the  total  number  of  tanks  and  the  number 
that  are  available  according  to  tank  regiment. 

Please  take  a  few  moments  and  find  this  section  in  files 
3-7. 

Following  this  top  section  in  each  file,  an  individual 
section  for  each  tank  regiment  is  listed  with  availability 
and  maintenance  information  on: 

1.  Tank  ID  code, 

2.  Front  location, 

3.  Maintenance  battalion  code. 
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4.  Number  of  battle  hours, 

5.  A  maintenance  code. 

Please  take  a  few  moments  and  look  through  files  3  -  7  to 
identify  this  information  and  review  possible  values. 

File  Number  8:  Tank  Division  Combat  Support 

This  file  contains  information  on  regiments  and 
battalions  that  support  the  Tank  Division  in  combat.  The 
units  listed  in  this  file  are: 

1.  Artillery  Regiment  (Section  1.1. 4.1), 

2.  Multiple  Rocket  Launcher  Battalion 

(Section  1.1. 4. 2), 

3.  Anti-Aircraft  Gun  Regiment  (Section  1.1. 4. 3), 

4.  Reconnaissance  Battalion  (Section  1.1. 4. 4). 

Each  of  these  sections  have  information  in  this  file  on: 

1.  Mission, 

2.  Subgroups, 

3.  Operational  capabilities. 

Please  take  a  few  Moments  and  browse  the  Tank  Division 
Comoat  Support  file. 

You  should  note  that  this  Tank  Division  Combat  Support 
file  (number  8)  contains  verbal  material  much  like  the  Army 
Operations  file  (number  1).  This  information  is  much 
different  than  the  statistics  and  tabled  data  on  tank 
equipment  and  inventory  (files  2-7). 

File  Number  9:  Motorized  Rifle  Equipment  Statistics 

This  file  includes  specific  and  miscellaneous  information 
on  the  following  armored  personnel  carriers  (APCs): 

1.  BMP-1  Armored  Personnel  Carrier  (Section  1.2. 1.1), 

2.  BTR-60  Armored  Personnel  Carrier  (Section  1.2. 1.2), 

3.  BTR-50P  Armored  Personnel  Carrier  (Section  1.2. 1.3), 

4.  BTR-152  Armored  Personnel  Carrier  (Section  1.2. 1.4;, 

5.  MT-L3  Multipurpose  Tracked  Vehicle  (Section  1.2. 1.5). 

On  each  of  these  armored  personnel  carriers  the  specific 
characteristics  that  may  be  included  are: 

1.  Crew, 

2.  Weight, 

3.  Size, 


207 


4.  Road  speed  and  range, 

5.  Armament, 

6.  Method  of  ranging, 

7.  Ammunition. 


In  addition,  notice  that  the  highlighted  lines  in  the 
Motorize.-  Rifle  Equipment  Statistics  file  access  subfiles 
associated  with  the  availability  and  inventories  of  each 
type  of  armored  personnel  carrier. 

Please  take  a  few  moments  and  look  over  the  Mov  rized 
Rifle  Equipment  Statistics  file  (number  9). 


You  should  note  that  this  Motorized  Rifle  Equipment 
Statistics  file  (file  number  9)  is  structured  and  contains 
similar  information  to  that  of  the  Tank  Equipment  Statistics 
file  (number  2 ) . 

File  Numbers  10  -  14:  Armored  Personnel  Carrier 
Inventory  and  Availability 

These  files  contain  information  about  the  availability 
and  inventories  of  each  armored  personnel  carrier  listed  in 
Motorized  Rifle  Equipment  Statistics  (file  number  9). 

At  the  top  of  each  of  these  files  is  a  section  containing 
information  about  the  total  number  of  armored  personnel 
carriers  and  the  number  that  are  available  according  to 
regiment . 

Please  take  a  few  moments  and  find  this  section  in  files 
10  -  14. 


Following  this  top  section  in  each  file,  an  individual 
section  for  each  motorized  rifle  regiment  is  listed  with 
availability  and  maintenance  information  on: 


1.  Armored  personnel  carrier  (APC)  or 
multipurpose  tracked  vehicle  (MTV)  ID  cod  , 

2.  Front  location, 

3.  Maintenance  battalion  code, 

4.  Number  of  battle  hours, 

5.  A  maintenance  code. 


Please  take  a  few  moments  and  look  through  files  10  -  14 
to  identify  this  information  and  review  possible  values. 

You  should  note  that  these  Motorized  Rifle  Inventory 
files  are  structured  and  contain  similar  tabular  information 
when  compared  to  the  Tank  Inventory  files  (numbers  3-7). 
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File  Number  15:  Motorized  Rifle  Division  Combat  Support 

This  file  contains  information  on  regiments  and 
battalions  that  support  the  Motorized  Rifle  Division  in 
combat.  The  units  listed  in  this  file  are  divided  into 
those  that  provide  direct  combat  support  and  those  that 
provide  combat  service  support. 

The  Motorized  Rifle  regiments  and  battalions  that  provide 
direct  combat  support  are  the: 

1.  Artillery  Regiment  (Section  1.2. 4. 1.1), 

2.  Anti-tank  Missile  Battalion  (Section  1.2. 4. 1.2), 

3.  Surface  to  Air  Missile  Regiment  (Section  1.2. 4. 1.3), 

4.  Signal  Battalion  (Section  1.2. 4. 1.4), 

5.  Engineer  Battalion  (Section  1.2. 4. 1.5). 

Please  take  a  moment  and  locate  these  sections  in  the 

Motorized  Rifle  Combat  Support  file. 

The  Motorized  Rifle  Battalions  that  provide  combat 

service  support  support  are  the: 

1.  Maintenance  Battalion  (Section  1.2. 4. 2.1), 

2.  Medical  Battalion  (Section  1.2. 4. 2. 2), 

3.  Motor  Transport  Battalion  (Section  1.2. 4. 2. 3). 

Please  take  a  moment  and  locate  these  sections  in  the 

Motorized  Rifle  Combat  Support  file  (number  15). 

Much  like  the  Army  Operations  file  (number  1)  and  the 
Tank  Division  Combat  Support  file  (number  8),  all  sections 
of  this  file  have  information  on: 

1.  Mission, 

2.  Subgroups, 

3.  Operational  capabilities. 

Please  take  another  look  and  locate  these  sections  in  the 
Motorized  Rifle  Combat  Support  file  (number  15). 

You  have  completed  the  training  on  the  information 
retrieval  database.  Please  take  a  5  minute  break. 


Appendix  F 

TUTORIAL  FOR  THE  INFORMATION  RETRIEVAL  SOFTWARE 


Information  Retrieval  Training 

The  purpose  of  this  training  session  is  to  familiarize 
you  with  the  basic  procedures  of  the  experiment.  In  this 
first  training  session  you  will  be  introduced  to: 

1.  the  primary  and  secondary  displays, 

2.  the  touch  keypad, 

3.  the  information  retrieval  task, 

4.  target  location. 

If  you  have  any  questions  during  this  training  session, 
please  do  not  hesitate  to  ask  the  experimenter. 

Primary  and  Secondary  Displays 

The  experiment  will  use  two  CRT  screens.  The  first  CRT 
screen  to  the  right  is  the  primary  display  and  will  be  used 
for  presenting  the  information  retrieval  system.  The  second 
CRT  screen  to  the  right  is  the  secondary  display  and  will  be 
used  for  presenting  target  information  to  you. 

The  primary  display  contains  the  information  retrieval 
environment.  This  information  retrieval  display  has  several 
important  areas  in  it.  Of  major  importance  is  the  7-line 
window  which  is  defined  by  two  horizontal  lines.  This 
7- line  window  is  used  to  display  the  different  computer 
files  in  the  information  retrieval  system. 

Associated  with  the  7-line  window  are  line  numbers  which 
are  presented  to  the  left  of  the  computer  file.  You  will 
have  to  refer  to  these  line  numbers  when  you  locate  a  target 
in  a  file. 

Directly  above  the  7-line  window,  is  a  line  that  displays 
the  name  and  length  of  the  current  computer  file.  As  can  be 
seen,  the  current  file  is  called  Army  Operations  and  is  100 
lines  long. 
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The  line  below  the  7-line  window  is  an  input  and  message 
line.  If  a  search  procedure  requires  input,  the  data  you 
enter  will  be  displayed  there.  In  addition,  messages  will 
be  displayed  in  the  input  area  when  you  use  the  search 
procedures . 

On  the  lower  portion  of  the  primary  display  is  the  touch 
keypad  and  the  work  area.  The  touch  keypad  is  used  for 
selecting  search  procedures  and  will  be  described  in  the 
next  section  of  instructions.  The  work  area  on  the  lower 
left  hand  side  of  the  primary  display  will  be  used  to 
display  information  associated  with  specific  search 
procedures . 

You  will  select  search  procedures  on  the  keypad  displayed 
in  the  lower  right  hand  corner  of  the  primary  display,  much 
like  you  have  been  doing  on  this  display  when  asking  for 
more  information.  Touching  a  key  will  activate  a  search 
procedure.  This  input  device  is  known  as  a  touch  panel. 

This  specific  touch  panel  uses  horizontal  and  vertical 
light  emitting  diodes  to  direct  infra-red  light  beams  across 
the  surface  of  the  primary  display.  When  a  horizontal  and 
vertical  beam  are  broken  the  touch  panel  sends  a  code  to  the 
computer  indicating  the  search  procedure  you  selected. 

At  the  present  time,  the  touch  keypad  is  not  working. 
However,  soon  you  will  be  given  practice  using  it  for  search 
procedure  selection. 


Information  Retrieval 

As  described  earlier,  this  system  is  a  information 
retrieval  environment  with  files  arranged  hierarchically. 
At  the  start  of  every  trial  you  will  be  positioned  at  the 
top  file  in  the  hierarchy. 

This  can  be  seen  on  the  primary  display.  The  top  file  in 
the  hierarchy  is  the  Army  Operations  file.  This  file 
contains  general  information  about  a  hypothetical  enemy  and 
highlighted  lines  where  more  specific  enemy  data  can  be 
retrieved. 

For  example,  the  primary  display  currently  contains  the 
highlighted  line.*  Tank  Equipment  Statistics.  This  indicates 
that  more  information  can  be  retrieved  on  tank  equipment. 
The  search  procedures  for  accessing  this  information  will  be 
explained  later. 
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During  training  and  practice  trials,  it  may  be  useful  to 
use  the  map  of  the  file  hierarchy  that  was  provided  earlier. 
This  map  will  be  available  on  all  training  and  practice 
trials.  However,  it  will  not  be  available  during  the  actual 
experiment.  Therefore,  do  not  rely  upon  it  as  an  aid,  but 
as  a  training  device  for  learning  the  file  hierarchy. 

Target  Location 

On  each  trial  in  this  experiment,  you  will  be  asked  to 
find  specific  information.  This  information  will  be  shown 
on  the  secondary  display. 

Once  you  have  located  the  target  information  using  the 
search  procedures  on  the  primary  display,  you  should  press 
the  TARGET  key.  This  key  is  located  on  the  keyboard  of  the 
primary  -display.  The  computer  will  then  then  prompt  you  to 
enter  the  line  number  of  the  target.  The  prompt  "Target 
line  number:"  will  appear  on  the  input  and  message  line. 

The  line  number  of  the  target  should  then  be  typed  using 
the  digit  keys  on  the  main  keyboard.  Then  the  RETURN  key 
must  be  pressed  to  enter  your  response. 

At  this  time  please  enter  the  target  number  shown  on  the 
secondary  display  by: 

1.  Pressing  the  TARGET  key, 

2.  Typing  the  line  number  with  the  digit  keys, 

3.  Pressing  the  RETURN  key. 

All  of  these  keys  are  on  the  primary  keyboard. 

If  you  have  any  questions  please  ask  the  experimenter. 

If  not  lets  start  learning  the  search  procedures. 

As  an  aid  to  learning  and  using  these  search  procedures, 
a  short  reference  sheet  on  the  methods  for  search  procedure 
use  is  available  now  and  throughout  the  experiment. 

SCROLL  UP  AND  DOWN 

The  scroll  up  and  down  procedures  move  the  lines  of  the 
computer  file  one  by  one  through  the  7-line  window.  The 
SCROLL  keys  are  located  at  the  top  of  the  touch  keypad. 
Pressing  either  of  the  two  SCROLL  keys  will  move  the  lines 
of  the  computer  file  continuously  through  the  display 
window.  For  example,  when  the  SCROLL  DOWN  key  is  pressed, 
the  movement  will  be  toward  the  last  line  in  the  file. 
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At  this  time,  please  use  the  SCROLL  UP  and  SCROLL  DOWN 
procedures  to  find  the  target  displayed  on  the  secondary 
display.  You  will  have  two  trials  to  practice  SCROLL  UP  and 
DOWN. 

(Practice  1:  Locate  the  number  of  officers  in  a  tank 
regiment . 

Practice  2:  Verify  that  the  tank  division  will  conduct 
pursuit  operations  against  a  withdrawing 
enemy. ) 

Now  lets  learn  about  the  page  procedures. 

PAGE  UP  AND  DOWN 

The  Page  Up  and  Page  Down  procedures  display  consecutive 
segments  of  the  computer  file  in  the  7-line  window.  The 
Page  procedure  is  selected  by  pressing  either  the  PAGE  UP  or 
PAGE  DOWN  keys,  located  at  the  top  of  the  touch  keypad. 

The  PAGE  UP  key  displays  the  next  segment  of  the  computer 
file  moving  towards  the  first  line  in  the  file.  The  PAGE 
DOWN  key  displays  the  next  segment  of  the  computer  file 
moving  towards  the  last  line  in  the  file. 

At  this  time,  please  use  the  PAGE  UP  and  PAGE  DOWN 
procedures  to  find  the  target  displayed  on  the  secondary 
display.  You  will  have  two  practice  PAGE  UP  and  PAGE  DOWN. 

(Practice  1:  Locate  the  number  of  officers  at  the  division 
headquarters  of  the  motorized  rifle  division. 

Practice  2:  Locate  the  number  of  enlisted  personnel  in  an 
anti-tank  missile  battalion.) 

Now  lets  learn  about  the  Section  procedure. 

SECTION 

The  Section  procedure  can  be  used  to  take  advantage  of 
the  hierarchical  nature  of  each  computer  file.  That  is,  the 
Section  procedure  will  display  a  table  of  contents  for  the 
current  computer  file. 

To  use  the  Section  procedure,  touch  the  SECTION  key  on 
the  second  line  of  the  touch  keypad.  The  prompt  "Section 
number:"  will  then  be  displayed  on  the  input  and  message 
line.  In  addition,  the  major  sections  of  the  current  file 
will  be  shown  in  the  work  area  of  the  primary  display. 
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A  section  can  be  typed  at  this  point  with  the  digit  keys 
at  the  top  of  the  primary  keyboard.  If  the  RETURN  key  is 
then  pressed,  the  Section  procedure  will  position  the 
display  window  at  the  line  where  the  section  begins. 

A  summary  of  these  procedures  for  positioning  the  display 
window  to  a  major  section  in  a  file  is: 

1.  Touch  SECTION  on  the  touch  keypad, 

2.  Select  and  type  the  section  number  with  the 
digit  keys  on  the  primary  keyboard, 

3.  Press  RETURN  on  the  primary  keyboard. 

At  this  time,  please  try  the  Section  procedure  to 
position  the  Army  Operations  file  to  the  section  on  the 
"Motorized  Rifle  Division".  When  you  locate  this  line 
please  enter  its  line  number  with  the  TARGET  key.  If  you 
have  any  difficulties,  please  ask  the  experimenter  for  help. 

In  addition  to  the  major  sections  that  can  be  displayed 
with  the  Section  procedure,  this  search  procedure  can  also 
be  used  to  present  specific  subsections  of  a  computer  file. 

To  select  and  display  subsections,  the  SECTION  key  must 
be  touched  and  a  major  section  number  typed  that  has 
subsections  available.  To  check  to  see  if  any  subsections 
are  available,  simply  look  for  a  "*"  by  the  section  number. 

To  present  these  subsections  you  must  touch  the  SECTION 
key  again.  Finally,  the  subsection  is  selected  by  typing 
the  subsection  number  and  pressing  the  RETURN  key. 

A  summary  of  these  procedures  for  positioning  the  display 
window  to  a  subsection  in  a  file  is: 

1.  Touch  SECTION  on  the  touch  keypad, 

2.  Select  and  type  the  section  number  that  have 
subsections  available, 

3.  Touch  SECTION  again, 

4.  Select  and  type  the  subsection  number, 

5.  Press  the  RETURN  key  on  the  primary  keyboard. 

At  this  time,  please  try  the  Section  procedure  to 
position  the  Army  Operations  file  to  the  subsection  on  "Tank 
Division  Combat  Support".  When  you  locate  this  line  number, 
please  enter  it  as  the  target  line.  Once  again,  if  you  have 
any  difficulties,  please  ask  the  experimenter  for  help. 
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When  using  the  Section  procedure  to  browse  sections  in 
the  file,  it  might  be  necessary  for  you  to  backup  and  choose 
a  different  section.  This  can  be  done  using  the  DELETE  key 
on  the  primary  keyboard.  Pressing  the  DELETE  key  will  erase 
any  entries,  and  if  there  is  a  semi-colon  the  DELETE  key 
will  backup  and  display  the  previous  set  of  sections. 

At  this  time,  please  use  the  Section  procedure  to  select 
the  subsections  of  the  "Tank  Division",  and  then  use  the 
DELETE  key  to  to  backup  and  fixid  information  on  "Motorized 
Rifle  Division  Mission"  When  you  are  finished,  enter  this 
line  number  as  the  target  line. 

Additional  Hints  on  the  Section  Procedure 

If  at  anytime  you  want  to  cancel  the  Section  procedure 
without  changing  the  position  of  the  file  in  the  7-line 
window,  just  touch  another  search  procedure  key. 

Also,  you  might  have  noticed  that  some  of  the  sections 
are  highlighted.  These  highlighted  section  lines  indicate 
that  more  specific  data  can  be  retrieved  from  additional 
files.  Therefore,  the  Section  procedure  can  be  useful  for 
finding  the  access  points  to  this  additional  information. 

Now,  we  would  like  you  to  use  the  Section  procedure  to 
find  the  target  on  the  secondary  display.  Please  refer  to 
the  search  procedure  reference  sheet  if  you  need  a  quick 
summary  of  the  methods  for  the  Section  procedure.  You  will 
have  one  additional  trial  after  this  target  to  practice  the 
Section  procedure. 

(Practice  1:  Verify  that  an  operational  capability  of 

the  tank  division  is  to  defend  in  place  in 
order  to  gain  time  for  massing  forces. 

Practice  2:  Verify  that  the  motorized  rifle  division  has 
a  tank  regiment  subgroups.) 

Now  lets  learn  about  the  Search  procedure. 

SEARCH 

The  Search  procedure  can  be  used  to  scan  the  current 
computer  file  for  a  piece  of  text.  The  text  can  include  any 
letter,  digit,  or  special  symbol  that  is  on  the  keyboard. 
However,  it  is  important  to  remember  that  a  blank  space  is 
also  recognized  as  a  character.  Therefore,  "tank"  and 
"  tank"  are  different  pieces  of  text.  In  addition,  the 
difference  between  upper  and  lower  case  letters  is  not 
recognized  by  the  Search  procedure. 
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To  use  the  Search  procedure  you  must  touch  the  SEARCH  key 
on  the  touch  keypad.  The  prompt  "Search  for:"  will  then  be 
displayed  on  the  input  and  message  line.  At  this  point  you 
can  type  a  piece  of  text  that  is  up  to  67  characters  long 
through  the  primary  keyboard.  If  you  make  a  mistake  when 
typing  the  text  use  the  DELETE  key  to  erase  all  or  part  of 
the  text . 

When  you  have  finished  typing  the  text,  press  the  RETURN 
key  to  start  the  Search  procedure.  The  Search  procedure 
will  attempt  to  find  all  occurrences  of  the  text  and  will 
store  up  to  10  locations  for  you  to  browse. 

To  browse  over  the  results  of  the  Search  procedure,  press 
the  BROWSE  key  on  the  keypad  to  the  right  of  the  primary 
keyboard.  Each  time  it  is  pressed  the  BROWSE  key  will 
position  the  7- line  window  at  a  new  occurrence  of  the  text. 

A  summary  of  the  methods  for  using  the  Search  procedure 

i  s : 

1.  Touch  SEARCH  on  the  touch  keypad, 

2 .  Type  the  text  on  the  primary  keyboard, 

3.  Press  RETURN  on  the  primary  keyboard, 

4.  Press  BROWSE  on  the  keypad  to  the  right  of  the 
primary  keyboard. 

At  this  time,  please  practice  using  the  Search  procedure 
to  find  the  target  on  the  secondary  display.  You  will  have 
two  trials  to  practice  the  Search  procedure.  If  you  have 
any  difficulties,  please  ask  the  experimenter  for  help. 

(Practice  1:  Locate  the  number  of  officers  at  the 
headquarters  of  a  tank  division. 

Practice  2:  Verify  that  a  defense  of  the  motorized 

rifle  division  is  to  defend  a  zone  in  the 
main  defensive  belt  of  the  combined 
arms  army. ) 

Now  lets  learn  about  the  Search-And  and  Search-And-Not 
procedures . 

SEARCH-AND/NOT 

The  Search-And,  and  the  Search-And-Not  procedures  are 
extensions  of  the  basic  Search  procedure  so  that  multiple 
pieces  of  text  can  be  searched  for.  The  Search-And 
procedure  allows  you  to  look  for  up  to  5  pieces  of  text 
within  7  lines  of  each  other.  The  Search-And-Not  procedure 


216 


allows  you  to  search  for  a  piece  of  text  without  the 
occurrence  of  other  text  within  7  lines. 

To  use  the  Search-And  procedure  you  can  touch  the  SEARCH 
AND  key  on  the  touch  keypad.  The  prompt  "Search  for:"  will 
then  be  displayed  on  the  input  and  message  line.  At  this 
point  you  can  type  the  first  piece  of  text. 

Subsequently,  the  other  pieces  of  text  that  you  wish  to 
look  for  in  combination  with  the  first  piece  of  text  can  be 
tvped  by  touching  the  SEARCH  AND  key  on  the  touch  keypad  and 
typing  the  text  with  the  primary  keyboard.  You  can  repeat 
these  steps  until  there  are  5  pieces  of  text. 

When  you  have  finished  entering  the  text,  press  the 
RETURN  key  to  start  the  Search-And  procedure.  The 
Search-And  procedure  will  store  up  to  10  locations  for  you 
to  scan  with  the  BROWSE  key. 

When  using  the  Search-And  procedure  the  DELETE  key  can  be 
used  for  erasing  AND  prompts  and  text. 

A  summary  of  these  procedures  for  using  the  Search-And 
procedure  is: 

1.  Touch  SEARCH  AND  on  the  touch  keypad, 

2.  Type  the  first  piece  of  text  using  the  primary 
keyboard, 

3 .  Touch  SEARCH  AND  on  the  touch  keypad, 

4.  Type  the  text  using  primary  keyboard, 

(Please  note  that  you  can  repeat  steps  3  and  4 

until  5  pieces  of  text  are  entered. ) 

5.  Press  RETURN  to  start  the  Search-And  procedure, 

6.  Press  BROWSE  to  view  the  locations  that  were 
found. 

At  this  time,  please  try  the  Search-And  procedure  to  find 
the  line  :  "Tank  Division  Operational  Capabilities".  When 
you  find  it,  enter  the  line  number  as  the  target  number.  If 
you  have  any  difficulties  please  ask  the  experimenter  for 
help . 

To  use  the  Search-And-Not  procedure,  first  touch  the 
SEARCH  AND  NOT  key  and  type  the  text  you  would  like  to 
search  for.  Then,  touch  the  SEARCH-AND-NOT  key  again  and 
type  the  text  that  the  Search  procedure  should  exclude. 

Once  the  text  is  entered,  the  RETURN,  BROWSE,  and  DELETE 
keys  on  the  keyboard  will  work  exactly  as  they  did  with  the 
Search-And  procedure. 
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The  Search-And-Not  procedure  can  also  be  used  with  the 
Search-And  procedure.  The  only  restriction  is  that  the  text 
to  be  excluded  must  be  entered  last. 

A  summary  of  these  procedures  for  using  the 
Search-And-Not  procedure  is: 

1.  Touch  SEARCH-AND-NOT  on  the  touch  keypad, 

2.  Type  the  first  piece  of  text  using  the  primary 
keyboard, 

3.  Enter  any  SEARCH-AND  prompts  and  text, 

4.  Touch  SEARCH-AND-NOT  on  the  touch  keypad, 

5.  Type  the  text  to  be  excluded  from  search, 

(Please  note  that  you  can  repeat  steps  4  and  5  until 
5  total  pieces  of  text  are  entered. ) 

6.  Press  RETURN  to  start  the  Search-And-Not  procedure, 

7.  Press  BROWSE  to  view  the  locations  that  were  found. 

At  this  time,  please  try  the  Search-And-Not  procedure  to 
find  information  on  "tank  division  without  information  on 
combat  support".  When  you  have  line  39  at  the  top  of  the 
display,  enter  that  line  as  the  target  number.  If  you  have 
any  difficulties,  please  ask  the  experimenter  for  help. 

Now,  we  would  like  you  to  use  the 

Search-And/Search-And-Not  procedures  to  find  the  target  on 
the  secondary  display.  Please  refer  to  the  search  procedure 
reference  sheet  if  you  need  a  quick  summary  of  the  methods 
for  the  Search-And/Search-And-Not  procedures.  You  will  have 
one  additional  trial  to  practice  the  Search-And/Search-Not 
procedures . 

(Practice  1:  Locate  the  number  of  officers  in  the  rocket 
launcher  battalions. 

Practice  2:  Locate  the  tank  regiment  subgroup  that  is  not 
in  motorized  rifle  division.) 

Now  lets  learn  about  the  Index  procedure. 

INDEX 

The  index  procedure  can  be  used  to  find  information  in 
the  current  computer  file  much  like  an  index  in  a  book  is 
used  to  find  special  topics.  To  use  the  Index  procedure, 
touch  the  key  on  the  third  line  of  the  touch  keypad.  The 
computer  will  then  generate  the  index  in  the  work  area  of 
the  primary  display.  This  index  contains  an  alphabetical 
list  of  topics  and  subtopics. 
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The  index  can  be  browsed  by  pressing  the  INDEX  UP  or 
INDEX  DOWN  keys  on  the  keypad  to  the  right  of  the  primary 
keyboard.  These  keys  move  the  index  through  the  work  area 
in  much  the  same  way  that  the  PAGE  UP  and  PAGE  DOWN  keys 
move  the  computer  file  through  the  7- line  window. 

At  this  point,  please  touch  the  INDEX  touch  key,  and  use 
the  INDEX  UP  and  INDEX  DOWN  keys  to  position  the  index  to 
the  major  topic  of  "Regiments". 

If  you  see  a  topic  that  you  want  locations  on,  all  you 
have  to  do  is  type  this  topic  next  to  the  "Index  topic:" 
prompt  and  touch  the  INDEX  key.  If  the  topic  you  requested 
was  a  major  topic  then  the  Index  procedure  will  ask  you  to 
enter  a  subtopic.  Otherwise,  the  Index  procedure  will 
locate  the  subtopic  and  ask  you  to  confirm  your  selection  by 
touching  the  INDEX  key. 

After  confirming  an  index  topic  the  Index  procedure  will 
erase  the  index  from  the  work  area  and  display  a  list  of 
line  numbers  for  the  topic  chosen.  You  can  then  use  the 
BROWSE  key  to  look  through  the  list  of  line  numbers  or  you 
can  enter  a  choice  number  of  a  location. 

Finally,  if  you  wish  to  return  to  the  index  from  the 
location  list,  just  press  RETURN. 

When  entering  the  topic  only  the  first  few  letters  which 
identify  the  topic  need  to  be  entered.  If  there  are 
multiple  subtopics  in  an  index,  then  the  major  topic  must  be 
selected  first. 

A  summary  of  these  procedures  for  selecting  a  topic  in 
the  index  is: 

1.  Touch  INDEX  on  the  touch  keypad, 

2.  Type  a  topic  with  the  primary  keyboard, 

3.  Touch  the  INDEX  key  again, 

4.  Confirm  the  topic  by  touching  INDEX  or  select  a 
subtopic, 

5.  BROWSE  or  select  a  choice  of  line  locations, 

6.  Press  RETURN  to  return  to  the  index. 

Now,  we  would  like  you  to  use  the  Index  procedure  to  find 
the  target  on  the  secondary  display.  You  will  have  one 
additional  trial  after  this  target  to  practice  the  Index 
procedure.  If  you  have  any  problems  please  ask  the 
experimenter  for  help. 

(Practice  1:  Locate  the  number  of  officers  in  a  surface 
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to  air  missile  regiment. 

Practice  2:  Verify  that  the  motorized  rifle  division 
has  a  subgroup  of  independent  tank 
battalions . ) 

Now  lets  learn  about  the  Zoom-In  and  Out  procedures. 

ZOOM-IN  AND  ZOOM-OUT 

The  Zoom-In  and  Zoom-Out  procedures  are  used  for  moving 
through  the  file  hierarchy.  The  Zoom-In  procedure  accesses 
additional  information  for  lines  in  the  current  file  which 
are  highlighted.  The  Zoom-Out  procedure  backs  up  to  the 
previous  file  in  the  hierarchy. 

To  use  the  Zoom-In  procedure  touch  the  ZOOM-IN  key  on  the 
touch  keypad  when  a  highlighted  line  appears  in  the  7-lin® 
window.  To  use  the  Zoom-Out  procedure  touch  the  ZOOM-Oi  ’ 
key  at  any  time. 

When  using  Zoom-In  and  Zoom-Out  the  top  line  of  the 
primary  display  may  be  useful  for  orienting  in  the  file 
hierarchy.  In  addition,  the  map  of  the  file  hierarchy  may 
be  useful  during  the  training  trials  to  learn  the 
hierarchical  file  structure.  However,  do  not  rely  to  a 
great  extent  on  this  map,  since  it  will  not  be  available 
during  the  actual  experiment. 

At  this  time,  please  use  the  Zoom-In  and  Zoom-Out 
procedures  to  find  the  target  shown  on  the  seco-.iary 
display.  Please  note  that  the  Scroll  Up  and  Down  procedures 
are  active  to  facilitate  your  exploration  of  the  file 
hierarchy. 

(Practice  1:  Locate  the  total  number  of  available  T-72 
tanks  in  the  Bravo  regiment. 

Practice  2:  Verify  that  the  artillery  regiment's 
mission  is  to  provide  fire  support  to 
the  tank  divisions  making  a  main  advance. 

HINT:  look  in  the  combat  support  files.) 

Now  lets  learn  about  the  File  Select  procedure. 

FILE  SELECT 

The  File  Select  procedure  allows  you  to  browse  the  file 
hierarchy  before  loading  the  actual  computer  files.  The 
File  Select  procedure  works  very  much  like  the  Section 
procedure,  except  that  the  File  Select  procedure  displays 
the  structure  of  the  file  hierarchy. 
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To  use  the  File  Select  procedure,  touch  the  FILE  SELECT 
key  on  the  touch  keypad.  The  prompt  "File  number.-"  will 
then  be  displayed  on  the  input  and  message  line.  In 
addition,  other  files  that  can  be  accessed  through  the 
current  file  will  be  shown  in  the  work  area  of  the  primary 
splay. 

A  file  can  be  selected  at  this  point  with  the  digit  keys 
at  the  top  of  the  primary  keyboard.  If  the  RETURN  key  is 
then  pressed,  the  file  you  selected  will  be  loaded  into  the 
7-line  window. 

A  summary  of  these  procedures  for  browsing  and  choosing 
another  file  from  the  current  file  is: 

1.  Touch  FILE  SELECT  on  the  touch  keypad, 

2.  Select  and  type  the  file  number  with  the  digit 
keys  on  the  primary  keyboard, 

3.  Touch  RETURN  on  the  primary  keyboard. 

At  this  time,  please  try  the  File  Select  procedure  to 
load  the  file  on  "Motorized  Rifle  Division  Combat  Support". 
When  you  get  to  this  file,  please  enter  line  number  1  as  the 
target  number.  If  you  have  any  difficulties,  please  ask  the 
experimenter  for  help. 

In  addition  to  browsing  and  choosing  files  that  are  just 
below  the  current  file  in  the  hierarchy,  the  File  Select 
procedure  can  also  browse  and  select  fil^s  that  are  more 
than  one  level  below  the  current  file. 

To  select  and  display  these  files  touch  the  FILE  SELECT 
key.  Then  check  to  see  if  there  are  any  subfiles  available 
by  looking  for  a  "*"  by  a  file  number.  At  this  point, 
select  a  file  that  you  would  like  to  see  subfiles  on,  and 
type  the  file  number  using  the  digit  keys. 

Next,  co  present  these  subfiles  you  must  touch  the  FILE 
SELECT  key  again.  Finally,  one  of  these  subfiles  can  be 
loaded  into  the  7- line  window  by  typing  the  file  number  and 
pressing  the  RETURN  key. 

A  summary  of  these  procedures  for  browsing  and  selecting 
a  file  that  is  more  than  one  level  below  the  current  file  in 
the  hierarchy  is: 

1.  Touch  FILE  SELECT  on  the  touch  keypad, 

2.  Select  and  type  the  file  number  that  has  subfiles 
available, 

3.  Touch  FILE  SELECT  again. 


221 


4.  Select  and  type  the  subfile  number, 

5.  Press  the  RETURN  key  on  the  primary  keyboard. 

At  this  time,  please  try  the  File  Select  procedure  to 
load  a  file  on  the  "PT-76  Tank  Inventory  and  Availability". 
When  you  get  to  this  file,  enter  line  number  1  as  the  target 
number.  If  you  have  any  difficulties,  please  ask  the 
experimenter  for  hexp. 

When  using  the  File  Select  procedure  to  browse  the  file 
hierarchy,  it  might  be  necessary  for  you  to  backup  and 
choose  a  different  file.  This  can  be  done  using  the  DELETE 
key  on  the  primary  keyboard.  Pressing  the  DELETE  key  will 
erase  any  entries,  and  if  there  is  a  semi-colon  the  DELETE 
key  will  backup  and  display  the  previous  set  of  files. 

At  this  time,  please  use  the  File  Select  procedure  to 
select  subfiles  for  "Tank  Equipment  Statistics".  Then  use 
the  DELETE  key  to  backup  and  load  the  motorized  rifle  file 
"BMP-1  Inventory  and  Availability".  When  you  get  to  this 
file,  enter  line  number  1  as  the  target  number. 

Additional  Hints  on  the  File  Select  Procedure 

It  at  anytime  you  want  to  cancel  the  File  Select 
procedure  without  loading  a  new  file  in  the  7-line  window, 
just  touch  another  search  procedure  key. 

Also,  you  might  have  noticed  that  File  Select  procedures 
can  only  access  the  files  that  are  below  the  current  file  in 
the  hierarchy.  If  you  want  to  view  a  larger  portion  of  the 
file  hierarchy,  it  will  be  necessary  to  backup  using  the 
Zoom-Out  procedure. 

Finally,  when  using  File  Select  it  may  be  useful  to  refer 
to  the  line  above  the  7-line  window  and  the  map  of  the  file 
hierarchy.  However,  please  use  the  map  for  training 
purposes  only,  since  it  will  not  be  available  during  the 
actual  experiment. 

Now,  we  would  like  you  to  use  the  File  Select  procedure 
to  find  the  target  on  the  secondary  display.  Please  refer 
to  the  search  procedure  reference  sheet  if  you  r.eed  a  quick 
summary  of  the  methods  for  the  File  Select  procedure.  The 
Scroll  procedures  are  available  for  movement  through 
individual  files  on  this  and  the  next  practice  trial. 

(Practice  1:  Verify  that  the  Crater  regiment  of  the 
motorized  rifle  division  has  59  total 
BTR-50P  armored  personnel  carriers  (APCs). 
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Practice  2:  Locate  the  size  of  the  T-72  tank  crew.) 

That  completes  the  first  training  session.  Please  see 
the  experimenter. 


Appendix  G 

SEARCH  PROCEDURE  REFERENCE  SHEET 


SEARCH  PROCEDURE  REFERENCE  SHEET 


SCROLL  UP  AND  SCROLL  DOWN 


Touch  SCROLL  UP  or  SCROLL  DOWN  keys  to  move  the  current  file  line  by 
line  through  the  window. 


PACE  UP  AND  PACE  DOWN 


Touch  PAGE  UP  or  PAGE  DOWN  keys  to  move  the  current  file  7  lines  at  a 
time  through  the  window. 


SECTION 


To  move  the  current  file  to  a  major  section: 

1.  Touch  SECTION  key, 

2.  Type  section  number, 

3.  Press  RETURN. 

To  move  the  current  file  to  a  subsection: 

1.  Touch  SECTION  key, 

2.  Type  section  number  that  has  subsections, 

3.  Touch  SECTION  key, 

4.  Type  subsection  number, 

5.  Press  RETURN. 

Use  the  DELETE  key  to  backup  and  to  erase  entries. 
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SEARCH 


To  Search  for  a  piece  of  text  in  the  current  file: 

1.  Touch  SEARCH  key, 

2.  Type  text, 

3.  Press  RETURN, 

4.  Press  BROWSE. 

Use  OELETE  key  to  remove  unnecessary  text. 


SEARCH- AMD 


To  search  for  multiple  pieces  of  text  within  7  lines  of  each  other: 

1.  Touch  SEARCH  AND  key, 

2.  Type  first  piece  of  text, 

3.  Touch  SEARCH  AND  key, 

4.  Type  text, 

(Steps  3  and  4  can  be  repeated  until  5  pieces  of  text  exist.) 

5.  Press  RETURN. 

6.  Press  BROWSE. 

Use  DELETE  to  erase  ANO  prompts  and  text. 


SEARCH-AND-NOT 


To  search  for  multiple  pieces  of  text  within  7  lines  of  each  other  excluding 
some  text: 

1.  Touch  SEARCH  ANO  NOT  key. 

2.  Type  first  piece  of  text. 

3.  Enter  any  Search-And  prompts  and  text. 

4.  Touch  SEARCH  AND  NOT  key, 

5.  Type  text  to  be  excluced, 

(Steps  4  and  S  can  be  repeated  until  S  total  pieces  of  text  exist.) 

5.  :-ess  RE- URN, 

7.  Press  BROWSE. 

Use  DELETE  to  erase  AND,  AND-NOT  prompts  and  text. 


s 
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INDEX 


To  select  a  topic  in  the  index: 

1.  Touch  INDEX, 

2.  Type  a  topic, 

3.  Touch  INDEX, 

4.  Confirm  the  topic  by  touching  INDEX  or  select  a  subtopic. 

5.  Press  BROWSE  or  enter  choice, 

6.  Press  RETURN  to  return  to  index. 

Use  INDEX  UP  and  INDEX  DOWN  on  the  keyboard  to  scan  Index. 

When  entering  topics  only  the  first  few  letters  which  identify  the  topic 
need  to  entered. 

if  there  are  multiple  subtopics  in  an  index  then  the  major  topic  must 
be  selected  first. 


ZOOM-IN 


Touch  the  ZOOM-IN  key  to  load  a  file  associated  with  a  highlighted  line  in  the 
window. 


ZOOM-OUT 


Touch  ZOOM-OUT  anytime  to  back  up  to  the  previous  file  in  the  hierarchy. 


FILE  SELECT 


To  browse  and  choose  files  from  the  current  file: 

1.  Touch  FILE  SELECT  key, 

2.  Type  file  number, 

3.  Press  RETURN. 

To  bro%\se  and  select  *iles  more  than  one  level  below  current  file: 

1.  Touch  FILE  SELECT  key, 

2.  Type  file  number  that  has  subfiles, 

3.  Touch  FILE  SELECT  key, 

4.  Type  subfile  number, 

5.  Press  RETURN. 

Use  the  DELETE  key  to  backup  and  to  erase  entries. 

Use  ZOOM-OUT  to  view  a  larger  portion  of  file  hierarchy. 
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Appendix  H 

SEARCH  PROBLEMS  USED  IN  THE  EXPERIMENT 


ITEM  CLUSTER 


SEARCH  PROBLEM 


1  2 


2  3 

3  7 

4  10 


5  9 


6  1 


7  1 


Locate  the  battle  hours  of  the  PT-76  tank  with 
the  Blue-5  maintenance  battalion  and  a  Waterjet 
maintenance  code. 

Locate  the  maintenance  battalion  of  the  armored 
personnel  carrier  with  an  ID  code  of  69024. 

Locate  the  range  of  the  armored  personnel 
vehicle  that  does  not  have  an  armament. 

Verify  that  an  operational  capability  of  the 
multiple  rocket  launcher  battalion  is  to 
provide  counter-bombardment. 

Verify  that  there  is  a  long  range 
reconnaissance  company  that  functions  as  a 
subgroup  of  the  reconnaissance  battalion. 

Locate  the  BMP-1  vehicle  that  has  a  Searchlight 
maintenance  code  and  the  largest  number  of 
battle  hours. 

Verify  that  the  BMP-1  armored  personnel  carrier 
number  6643A  has  139  battle  hours. 


8  7  Locate  the  crew  size  of  the  armored  personnel 

carrier  that  carries  the  largest  number  of 
infantry. 

9  5  Locate  the  number  of  vehicles  in  the  Razor 

regiment. 

10  11  Locate  the  number  of  enlisted  personnel  in  the 

maintenance  battalion. 


11  1  Locate  the  BTR-60  armored  personnel  carrier 

that  has  a  Hydro jet  maintenance  code  and  is  not 
on  reserve. 
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ITEM  CLUSTER 


SEARCH  PROBLEM 


12  9 

13  10 

14  1 

15  12 

16  3 

17  1 

18  8 

19  7 

20  7 


Verify  that  the  service  battery  is  a  subgroup 
of  the  multiple  rocket  launcher  battalion. 

Locate  the  first  operational  capability  of  the 
maintenance  battalion. 

Verify  that  the  MT-LB  with  the  ID  code  of 
4DF6CD  is  stationed  in  the  East. 

Verify  that  the  motorized  rifledlO  division 
attack  as  a  second  echelon  division. 

Locate  the  ID  code  of  the  armored  personnel 
carrier  that  has  a  range  of  400km,  a 
maintenance  code  of  Track,  and  a  Yellow-2 
maintenance  battalion. 

Locate  the  MT-LB  vehicle  that  has  a  Engine 
maintenance  code  and  is  not  stationed  in  the 
East,  West,  or  South. 

Locate  the  crew  of  the  tank  with  a  length  of 
6.715m,  a  width  of  3.352m  and  a  height  of  2.4m. 

Locate  the  height  of  the  BMP-1  armored 
personnel  carrier. 

Verify  that  the  main  armament  of  the  MT-LB 
tracked  vehicle  is  turret  mounted. 


21  10 

22  9 

23  1 

24  8 

7 


Verify  that  the  anti-aircraft  gun  regiment 
operates  as  a  part  of  march  column. 

Locate  the  only  subgroup  of  the  surface  to  air 
missile  regiment. 

Verify  that  the  fifth  BTR-60  armored  personnel 
carrier  in  the  Echo  regiment  is  stationed  in 
the  South. 

Verify  that  the  T-62  vehicle  can  lay  its  own 
smoke  column. 

Locate  the  possible  calibers  of  the 
guns  that  are  mounted  on  the  BTR-50P. 
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machine 
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ITEM  CLUSTER  SEARCH  PROBLEM 


26  9 

27  2 

28  4 

29  4 

30  1 

31  8 

32  2 

33  9 

34  3 

35  5 

36  12 

37  8 

38  10 

39  7 

8 


Locate  the  only  subgroup  of  the  anti-aircraft 
gun  regiment. 

Verify  that  the  T-64  tank  in  the  Bullet 
regiment  with  a  Track  maintenance  code  is 
serviced  by  the  Yellow-4  maintenance  battalion. 

Verify  that  the  heavy  tank  with  ID  code  343X  is 
on  reserve. 

Locate  the  tank  with  ID  code  370P. 

Locate  the  BMP-1  armored  personnel  carrier  in 
the  Harrow  regiment  that  is  on  reserve  and  has 
the  fewest  battle  hours. 

Verify  that  the  T-72  tank  has  a  water  cooled 
diesel  engine. 

Locate  the  fourth  tank  in  the  T-72  Spearhead 
regiment. 

Verify  that  anti-tank  battalions  are  subgroups 
of  the  artillery  regiment  which  supports  the 
tank  division. 

Locate  the  vehicle  with  ID  code  34037  that  is 
on  reserve  and  has  205  battle  hours. 

Verify  that  there  are  14  available  armored 
personnel  carriers  in  the  Lion  regiment. 

Verify  that  the  motorized  rifle  division  will 
sometimes  operate  as  a  front  reserve. 

Locate  the  main  armament  of  the  T-72  tank. 

Verify  that  an  operational  capability  of  the 
reconnaissance  battalion  is  to  pinpoint  key 
terrain. 

Locate  the  secondary  armament  of  the  vehicle 
with  the  heavy  machine  gun. 

Locate  the  speed  of  the  tank  with  the  smallest 
range. 
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ITEM 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 


CLUSTER  SEARCH  PROBLEM 

7  Verify  that  the  BTR-60  is  believed  to  be 

inadequate  for  hard  driving  over  rough  ground. 

7  Locate  the  amphibious  speed  of  the  armored 

personnel  carrier  that  can  climb  a  gradient  of 
30  degrees  and  is  propelled  by  twin  hydrojets. 

2  Verify  that  the  Arrow  regiment  has  18  T-72 

tanks  available. 

8  Locate  the  firing  rate  of  the  T-10. 

1  Locate  the  BTR-152  armored  personnel  carrier 

with  a  Wheel  maintenance  code  in  the  Taurus 
regiment. 

2  Locate  the  T-10  vehicle  with  the  most  battle 
hours . 

1  Locate  the  BTR-152  armored  personnel  carrier 

that  is  stationed  in  the  South  with  the  ID  code 
of  C971 . 

1  Locate  the  MT-LB  that  has  110  battle  hours  and 
does  not  have  the  Blue-1  maintenance  battalion. 

7  Locate  the  road  range  of  the  vehicle  with 

infra-red  headlamps  and  viewers. 

3  Locate  the  vehicle  that  can  travel  at  80km/hr 
and  has  an  ID  code  of  9402B. 

2  Locate  the  T-72  tank  in  the  regiment  number  890 
that  has  a  maintenance  code. 

3  Locate  the  armored  personnel  carrier  with  1 

battle  hour  that  has  a  speed  of  43  km/hr  and  a 
range  of  250km. 

12  Verify  that  there  are  motorized  rifle  regiments 
in  the  motorized  rifle  division. 

6  Locate  the  tank  regiment  that  has  10  35-ton 

tanks  in  total. 

6  Locate  the  total  number  of  available  37.5  ton 
battle  tanks  in  regiment  103. 
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ITEM  CLUSTER  SEARCH  PROBLEM 


56 

3 

Locate  the  armored  personnel  carrier  with 
dimensions  7.56m  by  2.82m  by  2.31m  and  68 
battle  hours. 

57 

12 

Verify  that  the  tank  division  will  conduct 
mobile  counter-attack  operations. 

58 

10 

Verify  that  an  operational  capability  of  the 
reconnaissance  battalion  is  to  identify  routes 
for  advance  and  withdrawal  for  the  tank 
division. 

59 

1 

Locate  the  BTR-50P  armored  personnel  carrier 
with  300  battle  hours  that  is  not  on  reserve. 

60 

4 

Locate  the  T-62  tank  with  the  Track  maintenance 
code  that  has  the  largest  number  of  battle 
hours . 

61 

4 

Locate  the  battle  tank  with  ID  code  21XF. 

62 

7 

Locate  the  horsepower  of  the  BTR-152  armored 
personnel  carrier. 

63 

1 

Locate  the  possible  calibers  of  the  machine 
guns  that  are  mounted  on  the  BTR-50P. 

64 

4 

Locate  the  amphibious  tank  with  an  ID  code  of 
1942. 

65 

2 

Locate  the  T-62  tank  with  the  Track  maintenance 
code  that  has  the  largest  number  of  battle 
hours . 

66 

2 

Locate  the  T-62  tank  with  a  Yellow-6 

maintenance  battalion  stationed  in  the  West. 

67 

4 

Locate  the  tank  in  regiment  567  with  the  most 
battle  hours. 

68 

1 

Locate  the  BTR-60  armored  personnel  carrier 
with  the  Rocket  maintenance  code  that  is  in  the 
Echo  regiment. 

69 

9 

Locate  the  messenger  unit  subgroup  in  the 
combat  support  forces  of  the  motorized  rifle 
division. 

231 


ITEM  CLUSTER  SEARCH  PROBLEM 


70  5 

71  6 

72  1 


73  3 


74  5 

75  2 

76  9 

77  8 

78  8 

79  4 

80  2 

81  1 


82  10 


83  3 


Verify  that  the  Leopard  regiment  has  5  armored 
personnel  carriers  available. 

Locate  the  total  number  of  battle  tanks  in  the 
Renegade  regiment. 

Verify  that  the  fourth  MT-LB  armored  personnel 
carrier  in  regiment  1852  is  stationed  in  the 
Northwest. 

Locate  the  vehicle  with  a  crew  of  13  that  is  in 
the  Rover  regiment  and  has  an  ID  code  of 
4408E6 . 

Locate  the  total  number  of  armored  personnel 
carriers  in  regiment  number  499. 

Locate  the  T-62  tank  with  the  Fuel  maintenance 
code . 

Locate  the  third  subgroup  of  the  signal 
battalion. 

Verify  that  the  T-64  tank  does  not  have  an 
automatic  loader. 

Locate  the  width  of  the  light  amphibious  tank. 

Verify  that  the  tank  with  ID  code  7282  has  19 
battle  hours. 

Locate  the  T-62  tank  in  the  Scorpion  regiment 
with  a  Loader  maintenance  code. 

Locate  the  BTR-60  armored  personnel  carrier 
with  ID  code  4AAE3  that  is  in  the  Forest 
regiment. 

Verify  that  a  mission  of  the  artillery  regiment 
supporting  the  motorized  rifle  division  is  to 
assist  in  the  defense  of  critical  areas. 

Locate  the  maintenance  battalion  of  the  first 
armored  personnel  carrier  in  the  Sahara 
regiment.  Please  note  that  this  vehicle  carries 
ammunition. 
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84  10  Verify  that  the  engineer  battalion  can  operate 

a  sawmill. 

85  1  Verify  that  a  MT-LB  in  the  Whiskey  regiment  has 

4  battle  hours. 

86  2  Locate  the  T-10  tank  with  the  Orange-5 

maintenance  battalion  that  has  the  fewest 
battle  hours. 

87  4  Locate  the  tank  that  has  a  loader  as  a  crew 

member  and  a.i  ID  code  of  2P12. 

88  1  Locate  the  BTR-50P  armored  personnel  carrier 

that  has  a  Orange  maintenance  battalion,  a  Gun 
maintenance  code,  but  is  not  stationed  in  the 
East  or  North. 

89  8  Locate  the  method  of  ranging  for  the  T-10  tank. 

90  8  Locate  the  secondary  armament  of  the  tank  that 

uses  HVAP  ammunition  and  does  not  use  APFSDS 
ammunition. 

91  2  Locate  the  PT-76  tank  stationed  in  the  South 

with  a  Green-3  maintenance  crew  and  a  Turret 

maintenance  code. 

92  2  Locate  the  T-64  tank  with  the  Fuel  maintenance 

code . 

93  2  Locate  the  PT-76  tank  on  reserve  with  the  most 

battle  hours. 

94  11  Locate  the  number  of  enlisted  personnel  in  the 

medical  battalion. 

95  4  Verify  that  the  heavy  tank  with  the  ID  code 

2721  has  a  Loader  maintenance  code. 

96  11  Locate  the  number  of  enlisted  personnel  in  an 

anti-  aircraft  gun  regiment. 


Appendix  I 

INSTRUCTIONS  FOR  SORTING  SEARCH  PROBLEMS 


TARGET  SORTING  INSTRUCTIONS  -  PART  I 

On  the  index  cards  in  front  of  you  are  the  targets  that 
you  searched  for  during  days  3-6.  We  are  interested  in  how 
you  would  categorize  these  targets. 

Your  task  is  to  sort  these  cards  into  groups  that  are 
meaningful  to  you.  When  you  are  finished  please  write  a 
short  description  of  the  target  features  which  characterize 
the  group.  This  can  be  written  on  a  blank  index  card. 

In  addition,  please  supply  a  few  search  plans  for 
locating  this  information.  That  is,  please  list  the 
sequence  of  search  procedures  you  would  use  to  locate  each 
group  of  targets.  This  information  can  also  be  written  on  a 
blank  index  card.  The  search  procedure  reference  sheet  is 
available  if  you  need  to  know  the  names  and  functions  of  the 
search  procedures. 

Use  as  many  blank  index  cards  as  necessary  for  the 
descriptions  and  search  plans.  The  experimenter  will  answer 
any  questions  you  have  about  this  procedure.  When  you  are 
finished,  please  see  the  experimenter. 
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TARGET  SORTING  INSTRUCTIONS  -  PART  II 

Now  we  would  like  you  to  link  the  groups  of  targets  with 
the  larger  index  cards.  That  is,  use  the  larger  index  cards 
to  categorize  the  groups  from  Part  I  into  larger  and  more 
general  target  classes.  This  can  be  done  by  placing  the 
larger  index  card  on  top  of  the  target  groups  that  you 
believe  are  related.  Once  again,  please  write  on  the  large 
index  card  a  summary  of  each  target  class. 

The  experimenter  will  be  available  to  answer  any 
questions . 


Appendix  J 

ANALYSIS  OF  VARIANCE  SUMMARY  TABLES 


Experiment  1:  Subject  Groups  (Novice,  Experts)  by  Sessions 
(3-6) 


Dependent  Measure:  Total  Operations 


Source 

df 

SS 

F-ratio 

D-value 

Betv/een  Subiects 

GROUP  (G) 

1 

265.00 

6.04 

0.0224 

SUBJ/G 

2? 

965.76 

Within  Subiects 

SESSION  (S) 

3 

157.02 

11.07 

0.0001 

S*G 

3 

10.76 

0.76 

0.5246 

S* SUBJ/G 

66 

312.09 

TOTAL 

95 

1710.63 
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12  9 

13  10 

14  1 

15  12 

16  3 

17  1 

18  8 

19  7 

20  7 


Verify  that  the  service  battery  is  a  subgroup 
of  the  multiple  rocket  launcher  battalion. 

Locate  the  first  operational  capability  of  the 
maintenance  battalion. 

Verify  that  the  MT-LB  with  the  ID  code  of 
4DF6CD  is  stationed  in  the  East. 

Verify  that  the  motorized  rifledlO  division 
attack  as  a  second  echelon  division. 

Locate  the  ID  code  of  the  armored  personnel 
carrier  that  has  a  range  of  400km,  a 
maintenance  code  of  Track,  and  a  Yellow-2 
maintenance  battalion. 

Locate  the  MT-LB  vehicle  that  has  a  Engine 
maintenance  code  and  is  not  stationed  in  the 
East,  West,  or  South. 

Locate  the  crew  of  the  tank  with  a  length  of 
6.715m,  a  width  of  3.352m  and  a  height  of  2.4m. 

Locate  the  height  of  the  BMP-1  armored 
personnel  carrier. 

Verify  that  the  main  armament  of  the  MT-LB 
tracked  vehicle  is  turret  mounted. 


21  10 

22  9 

23  1 

24  8 

25  7 


Verify  that  the  anti-aircraft  gun  regiment 
operates  as  a  part  of  march  column. 

Locate  the  only  subgroup  of  the  surface  to  air 
missile  regiment. 

Verify  that  the  fifth  BTR-60  armored  personnel 
carrier  in  the  Echo  regiment  is  stationed  in 
the  South. 

Verify  that  the  T-62  vehicle  can  lay  its  own 
smoke  column. 

Locate  the  possible  calibers  of  the  machine 
guns  that  are  mounted  on  the  BTR-50P. 
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26  9 

27  2 


28  4 

29  4 

30  1 

31  8 

32  2 

33  9 

34  3 

35  5 

36  12 

37  8 

38  10 


39  7 

8 


SEARCH  PROBLEM 

Locate  the  only  subgroup  of  the  anti-aircraft 
gun  regiment. 

Verify  that  the  T-64  tank  in  the  Bullet 
regiment  with  a  Track  maintenance  code  is 
serviced  by  the  Yellow-4  maintenance  battalion. 

Verify  that  the  heavy  tank  with  ID  code  343X  is 
on  reserve. 

Locate  the  tank  with  ID  code  370P. 

Locate  the  BMP-1  armored  personnel  carrier  in 
the  Harrow  regiment  that  is  on  reserve  and  has 
the  fewest  battle  hours. 

Verify  that  the  T-72  tank  has  a  water  cooled 
diesel  engine. 

Locate  the  fourth  tank  in  the  T-72  Spearhead 
regiment. 

Verify  that  anti-tank  battalions  are  subgroups 
of  the  artillery  regiment  which  supports  the 
tank  division. 

Locate  the  vehicle  with  ID  code  34037  that  is 
on  reserve  and  has  205  battle  hours. 

Verify  that  there  are  14  available  armored 
personnel  carriers  in  the  Lion  regiment. 

Verify  that  the  motorized  rifle  division  will 
sometimes  operate  as  a  front  reserve. 

Locate  the  main  armament  of  the  T-72  tank. 

Verify  that  an  operational  capability  of  the 
reconnaissance  battalion  is  to  pinpoint  key 
terrain. 

Locate  the  secondary  armament  of  the  vehicle 
with  the  heavy  machine  gun. 

Locate  the  speed  of  the  tank  with  the  smallest 
range . 
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41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 
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7  Verify  that  the  BTR-60  is  believed  to  be 
inadequate  for  hard  driving  over  rough  ground. 

7  Locate  the  amphibious  speed  of  the  armored 

personnel  carrier  that  can  climb  a  gradient  of 
30  degrees  and  is  propelled  by  twin  hydrojets. 

2  Verify  that  the  Arrow  regiment  has  18  T-72 

tanks  available. 

8  Locate  the  firing  rate  of  the  T-10. 

1  Locate  the  BTR-152  armored  personnel  carrier 

with  a  Wheel  maintenance  code  in  the  Taurus 
regiment . 

2  Locate  the  T-10  vehicle  with  the  most  battle 
hours . 

1  Locate  the  BTR-152  armored  personnel  carrier 

that  is  stationed  in  the  South  with  the  ID  code 
of  C971 . 

1  Locate  the  MT-LB  that  has  110  battle  hours  and 
does  not  have  the  Blue-1  maintenance  battalion. 

7  Locate  the  road  range  of  the  vehicle  with 

infra-red  headlamps  and  viewers. 

3  Locate  the  vehicle  that  can  travel  at  80km/hr 
and  has  an  ID  code  of  9402B. 

2  Locate  the  T-72  tank  in  the  regiment  number  890 
that  has  a  maintenance  code. 

3  Locate  the  armored  personnel  carrier  with  1 

battle  hour  that  has  a  speed  of  43  km/hr  and  a 
range  of  250km. 

12  Verify  that  there  are  motorized  rifle  regiments 
in  the  motorized  rifle  division. 

6  Locate  the  tank  regiment  that  has  10  35-ton 

tanks  in  total. 

6  Locate  the  total  number  of  available  37.5  ton 
battle  tanks  in  regiment  103. 
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56 

57 

58 

59 

60 

61 

62 

63 

64 

65 

66 

67 

68 
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3  Locate  the  armored  personnel  carrier  with 

dimensions  7.56m  by  2.82m  by  2.31m  and  68 
battle  hours. 

12  Verify  that  the  tank  division  will  conduct 

mobile  counter-attack  operations. 

10  Verify  that  an  operational  capability  of  the 
reconnaissance  battalion  is  to  identify  routes 
for  advance  and  withdrawal  for  the  tank 
division. 

1  Locate  the  BTR-50P  armored  personnel  carrier 

with  300  battle  hours  that  is  not  on  reserve. 

4  Locate  the  T-62  tank  with  the  Track  maintenance 

code  that  has  the  largest  number  of  battle 

hours . 

4  Locate  the  battle  tank  with  ID  code  21XF. 

7  Locate  the  horsepower  of  the  BTR-152  armored 

personnel  carrier. 

1  Locate  the  possible  calibers  of  the  machine 
guns  that  are  mounted  on  the  BTR-50P. 

4  Locate  the  amphibious  tank  with  an  ID  code  of 

1942. 

2  Locate  the  T-62  tank  with  the  Track  maintenance 

code  that  has  the  largest  number  of  battle 

hours . 

2  Locate  the  T-62  tank  with  a  Yellow-6 

maintenance  battalion  stationed  in  the  West. 

4  Locate  the  tank  in  regiment  567  with  the  most 
battle  hours. 

1  Locate  the  BTR-60  armored  personnel  carrier 

with  the  Rocket  maintenance  code  that  is  in  the 
Echo  regiment. 

9  Locate  the  messenger  unit  subgroup  in  the 

combat  support  forces  of  the  motorized  rifle 
division. 
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70 

71 

72 

73 

74 

75 

76 

77 

78 

79 

80 

81 

82 

83 


CLUSTER  SEARCH  PROBLEM 

5  Verify  that  the  Leopard  regiment  has  5  armored 
personnel  carriers  available. 

6  Locate  the  total  number  of  battle  tanks  in  the 
Renegade  regiment. 

1  Verify  that  the  fourth  MT-LB  armored  personnel 
carrier  in  regiment  1852  is  stationed  in  the 
Northwest. 

3  Locate  the  vehicle  with  a  crew  of  13  that  is  in 
the  Rover  regiment  and  has  an  ID  code  of 
4408E6 . 

5  Locate  the  total  number  of  armored  personnel 

carriers  in  regiment  number  499. 

2  Locate  the  T-62  tank  with  the  Fuel  maintenance 
code . 

9  Locate  the  third  subgroup  of  the  signal 

battalion. 

8  Verify  that  the  T-64  tank  does  not  have  an 

automatic  loader. 

8  Locate  the  width  of  the  light  amphibious  tank. 

4  Verify  that  the  tank  with  J  code  7282  has  19 

battle  hours. 

2  Locate  the  T-62  tank  in  the  Scorpion  regiment 
with  a  Loader  maintenance  code. 

1  Locate  the  BTR-60  armored  personnel  carrier 

with  ID  code  4AAE3  that  is  in  the  Forest 
regiment. 

10  Verify  that  a  mission  of  the  artillery  regiment 
supporting  the  motorized  rifle  division  is  to 
assist  in  the  defense  of  critical  areas. 

3  Locate  the  maintenance  battalion  of  the  first 
armored  personnel  carrier  in  the  Sahara 
regiment.  Please  note  that  this  vehicle  carries 
ammunition. 
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85 

86 

87 

88 

89 

90 

91 

92 

93 

94 
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10  Verify  that  the  engineer  battalion  can  operate 
a  sawmill. 

1  Verify  that  a  MT-LB  in  the  Whiskey  regiment  has 
4  battle  hours. 

2  Locate  the  T-10  tank  with  the  Orange-5 
maintenance  battalion  that  has  the  fewest 
battle  hours. 

4  Locate  the  tank  that  has  a  loader  as  a  crew 
member  and  an  ID  code  of  2P12. 

1  Locate  the  BTR-50P  armored  personnel  carrier 
that  has  a  Orange  maintenance  battalion,  a  Gun 
maintenance  code,  but  is  not  stationed  in  the 
East  or  North. 

8  Locate  the  method  of  ranging  for  the  T-10  tank. 

8  Locate  the  secondary  armament  of  the  tank  that 

uses  HVAP  ammunition  and  does  not  use  APFSDS 
ammunition. 

2  Locate  the  PT-76  tank  stationed  in  the  South 

with  a  Green-3  maintenance  crew  and  a  Turret 

maintenance  code. 

2  Locate  the  T-64  tank  with  the  Fuel  maintenance 

code . 

2  Locate  the  PT-76  tank  on  reserve  with  the  most 

battle  hours. 

11  Locate  the  number  of  enlisted  personnel  in  the 
medical  battalion. 

4  Verify  that  the  heavy  tank  with  the  ID  code 
2721  has  a  Loader  maintenance  code. 

11  Locate  the  number  of  enlisted  personnel  in  an 
anti-  aircraft  gun  regiment. 
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TARGET  SORTING  INSTRUCTIONS  -  PART  I 

On  the  index  cards  in  front  of  you  are  the  targets  that 
you  searched  for  during  days  3-6.  We  are  interested  in  how 
you  would  categorize  these  targets. 

Your  task  is  to  sort  these  cards  into  groups  that  are 
meaningful  to  you.  When  you  are  finished  please  write  a 
short  description  of  the  target  features  which  characterize 
the  group.  This  can  be  written  on  a  blank  index  card. 

In  addition,  please  supply  a  few  search  plans  for 
locating  this  information.  That  is,  please  list  the 
sequence  of  search  procedures  you  would  use  to  locate  each 
group  of  targets.  This  information  can  also  be  written  on  a 
blank  index  card.  The  search  procedure  reference  sheet  is 
available  if  you  need  to  know  the  names  and  functions  of  the 
search  procedures. 

Use  as  many  blank  index  cards  as  necessary  for  the 
descriptions  and  search  plans.  The  experimenter  will  answer 
any  questions  you  have  about  this  procedure.  When  you  are 
finished,  please  see  the  experimenter. 
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TARGET  SORTING  INSTRUCTIONS  -  PART  II 

Now  we  would  like  you  to  link  the  groups  of  targets  with 
the  larger  index  cards.  That  is,  use  the  larger  index  cards 
to  categorize  the  groups  from  Part  I  into  larger  and  more 
general  target  classes.  This  can  be  done  by  placing  the 
larger  index  card  on  top  of  the  target  groups  that  you 
believe  are  related.  Once  again,  please  write  on  the  large 
index  card  a  summary  of  each  target  class. 

The  experimenter  will  be  available  to  answer  any 
questions . 


Appendix  J 

ANALYSIS  OF  VARIANCE  SUMMARY  TABLES 


Experiment  1:  Subject  Groups  (Novice,  Experts)  by  Sessions 
(3-6) 

Dependent  Measure:  Total  Operations 


Source 

df 

ss 

F-ratio 

o-value 

Between  Subiects 

GROUP  (G) 

1 

265.00 

6.04 

0.0224 

SUBJ/G 

22 

965.76 

Within  Subiects 

SESSION  (S) 

3 

157.02 

11.07 

0.0001 

S*G 

3 

10.76 

0.75 

0.5246 

S* SUBJ/G 

66 

312.09 

TOTAL 

95 

1710.63 
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Dependent  Measure: 

Different 

Operations 

Source 

df 

ss 

F-ratio 

p-value 

Between  Subiects 

GROUP  (G) 

1 

4.50 

2.95 

0.1002 

SUBJ/G 

22 

33.64 

Within  Subiects 

SESSION  (S) 

3 

2.04 

6.42 

0.0008 

S*G 

3 

0.12 

0.39 

0.7666 

S* SUBJ/G 

66 

7.01 

TOTAL 

95 

47.31 

Dependent  Measure: 

Total  Time 

Source 

df 

SS 

F-ratio 

n-value 

Between  Subiects 

GROUP  (G) 

1 

16315.56 

8.59 

0.0077 

SUBJ/G 

22 

41784.21 

Within  Subiects 

SESSION  (S) 

3 

14784.21 

20.73 

0.0001 

S*G 

3 

711.57 

1.01 

0.3930 

S* SUBJ/G 

66 

15426.31 

TOTAL 


95 


88771.50 
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Dependent  Measure: 

Total 

Movement 

d£ 

SS 

F-ratio 

D-value 

Between  Subiects 

GROUP  (G) 

1 

33488.01 

2.70 

0.1147 

SUBJ/G 

22 

273024.03 

Within  Subiects 

SESSION  (S) 

3 

20784.81 

4.91 

0.0040 

S*G 

3 

772.97 

0.18 

0.9057 

S* SUBJ/G 

66 

93099.97 

TOTAL 

95 

421169.80 
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Experiment  1:  Subject  Groups  (Slow  Novices  n  =  6,  Fast 
Novices  n  =  6,  Experts,  n  =  12)  by  Sessions  (3  -  6) 


Dependent  Measure: 

Total 

Operations 

Source 

df 

Type  HI  SS 

F-ratio 

o-value 

Between  Subiects 

GROUP  (G) 

2 

708.83 

14.26 

0.0001 

SUBJ/G 

21 

521.93 

Within  Subiects 

SESSION  (S) 

3 

163.19 

11.24 

0.0001 

S*G 

6 

17.99 

0.62 

0.7137 

S* SUBJ/G 

63 

304.85 

TOTAL 

95 

1710.63 

Dependent  Measure: 

Different  Operations 

• 

Source 

df 

Tyne  HI  SS 

F-ratio 

r>-value 

Between  Subiects 

GROUP  (G) 

2 

15.89 

7.50 

0.0035 

SUBJ/G 

21 

22.25 

Within  Subiects 

SESSION  (S) 

3 

2.02 

6.14 

0.0011 

S*G 

6 

0.22 

0.34 

0.9149 

S* SUBJ/G 

63 

6.91 

TOTAL 


95 


47.31 
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Dependent  Measure: 

Total 

Time 

Source 

df 

Type  III  SS 

F-ratio 

p-value 

Between  Subiects 

GROUP  (G) 

2 

42194.62 

27.86 

0.0001 

SUBJ/G 

21 

15905.16 

Within  Subiects 

SESSION  (S) 

3 

15123.36 

21.17 

0.0001 

S*G 

6 

1133.26 

0.79 

0.5788 

S* SUBJ/G 

63 

15004.63 

TOTAL 

95 

88771.50 

Dependent  Measure: 

Total 

Movement 

Source 

df 

Type  HI  SS 

F-ratio 

D-value 

Between  Subiects 

GROUP  (G) 

2 

138095.25 

8.61 

0.0019 

SUBJ/G 

21 

168416.80 

Within  Subiects 

SESSION  (S) 

3 

18633.02 

4.32 

0.0079 

S*G 

6 

3374.75 

0.39 

0.8818 

S* SUBJ/G 

63 

90498.19 

TOTAL 


95 


421169.80 
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Experiment  1:  Search  Strategy  Analyses  for  Slow  Novices 
(n  =  6),  Fast  Novices  (n  =  6),  and  Experts  (n  =  12) 


Dependent 

Measure : 

Scroll 

Up  Proportion 

Source 

df 

Tvoe  III  SS 

F-ratio 

p-value 

GROUP  (G) 

2 

0.1802 

4.57 

0.0225 

SUB J/G 

21 

0.4142 

TOTAL 

23 

0.5944 

Dependent 

Measure : 

Scroll 

Down  Proportion 

Source 

df 

Tvne  III  SS 

ErXatjp 

p-value 

GROUP  (G) 

2 

0.3925 

5.85 

0.0096 

SUBJ/G 

21 

0.7043 

TOTAL 

23 

1.0968 

Dependent 

Measure : 

Page  Up  Proportion 

Source 

df 

Type  III  SS 

F-ratio 

GROUP  (G) 

2 

0.0040 

0.14 

0.8678 

SUBJ/G 

21 

0.2963 

TOTAL 

23 

0.3003 

Dependent 

Measure : 

Page  Down  Proportion 

Source 

df 

Type  HI  SS 

F-ratio 

GROUP  (G) 

2 

0.0580 

0.48 

0.6270 

SUBJ/G 

21 

1.2751 

TOTAL 


23 


1.3331 
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Dependent  Measure:  Section  Proportion 


Source 

df 

Type  HI  SS 

F-ratio 

D-value 

GROUP  (G) 

2 

0.0685 

0.91 

0.4162 

SUBJ/G 

21 

0.7864 

TOTAL 

23 

0.8549 

Dependent 

Measure: 

Zoom 

In  Proportion 

Source 

M 

Type  III  SS 

F-ratio 

D-value 

GROUP  (G) 

2 

0.2177 

1.26 

0.3046 

SUBJ/G 

21 

1.8166 

TOTAL 

23 

2.0343 

Dependent 

Measure : 

Zoom 

Out  Proportion 

Source 

df 

Type  III  SS 

F-ratio 

GROUF  (G) 

2 

0.1064 

11.00 

0.0005 

SUBJ/G 

21 

0.1015 

TOTAL 

23 

0.2079 

Dependent 

Measure : 

File 

Select  Proportion 

Source 

df 

Type  III  SS 

F-ratio 

D-value 

GROUP  (G) 

2 

0.0479 

0.38 

0.6891 

SUBJ/G 

21 

1.3264 

TOTAL 


23 


1.3743 
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Dependent 

Measure:  Search 

Proportion 

Source 

df 

Tvoe  III  SS 

F-ratio 

o-value 

GROUP  (G) 

2 

0.0289 

0.33 

0.7223 

SUBJ/G 

21 

0.9201 

TOTAL 

23 

0.9491 

Dependent 

Measure:  Search- 

-And  Proportion 

Source 

df 

Tvoe  III  SS 

F-ratio 

wmmm 1 

GROUP  (G) 

2 

0.0060 

0.25 

0.7811 

SUBJ/G 

21 

0.2524 

TOTAL 

23 

0.2584 

Dependent 

Measure:  Search* 

-And-Not  Proportion 

Source 

df 

Type  III  SS 

F-ratio 

o-value 

GROUP  (G) 

2 

0.0002 

2.49 

0.1073 

SUBJ/G 

21 

0.0007 

TOTAL 

23 

0.0009 

Dependent 

Measure:  Index  Proportion 

Source 

df 

Tvoe  III  SS 

F-ratio 

GROUP  (G) 

2 

0.0511 

1.68 

0.2111 

SUBJ/G 

21 

0.3203 

TOTAL 


23 


0.3715 
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Experiment  1:  Subject  Groups  (Novice,  Experts)  by  Search 
Clusters  (1  -  12) 


Dependent  Measure: 

Total 

Operations 

Source 

df 

SS 

F-ratio 

Between  Subiects 

GROUP  (G) 

1 

1028.24 

4.75 

0.0403 

SUBJ/G 

22 

4762.85 

Within  Subiects 

CLUSTER  (C) 

11 

4849.97 

11.72 

0.0001 

C*G 

11 

351.83 

0.85 

0.5910 

C* SUBJ/G 

242 

9105.25 

TOTAL 

287 

20098.15 

Dependent  Measure: 

Different  Operations 

Source 

df 

SS 

F-ratio 

p-value 

Between  Subiects 

GROUP  (G) 

1 

13.52 

2.35 

0.1396 

SUBJ/G 

22 

126.58 

Within  Subiects 

CLUSTER  (C) 

11 

136.46 

16.31 

0.0001 

C*G 

11 

6.08 

0.73 

0.7138 

C* SUBJ/G 

242 

184.05 

TOTAL 


287 


466.68 
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Dependent  Measure: 

Total 

Time 

Source 

df 

£S 

£-  rati 9 

D-value 

Between  Subiects 

GROUP  (G) 

1 

60101.40 

5.71 

0.0258 

SUBJ/G 

22 

231456.09 

Within  Subiects 

CLUSTER  (C) 

11 

156832.10 

5.65 

0.0001 

C*G 

11 

28943.03 

1.04 

0.4089 

C* SUBJ/G 

242 

610331.88 

TOTAL 

287 

1087664.49 

Dependent  Measure: 

Total 

Movement 

Source 

df 

ss 

F-ratio 

D-value 

Between  Subiects 

GROUP  (G) 

1 

106960.42 

2.04 

0.1668 

SUBJ/G 

22 

1150897.79 

Within  Subiects 

CLUSTER  (C) 

11 

1396200.64 

11.06 

0.0001 

C*G 

11 

81985.85 

0.65 

0.7861 

C* SUBJ/G 

242 

2777475.94 

TOTAL 


287 


5513520.49 
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Experiment  1:  Subject  Groups  (Novice,  Experts)  by  Search 
Clusters  (1  -  12)  by  Search  Procedures 


Dependent  Measure:  Average  Polls 


Source 

df 

SS 

F-ratio 

n-value 

Between  Subiects 

GROUP  (G) 

1 

1.13 

2.35 

0.1396 

SUBJ/G 

22 

10.55 

Within  Subiects 

CLUSTER  (C) 

11 

11.37 

16.31 

0.0001 

C*G 

11 

0.51 

0.73 

0.7138 

C* SUBJ/G 

242 

15.34 

SEARCH  PROCEDURE 

(S)  11 

145.10 

30.25 

0.0001 

S*G 

11 

3.49 

0.73 

0.7135 

S* SUBJ/G 

242 

105.53 

C*S 

121 

54.41 

12.87 

0.0001 

C*S*G 

121 

6.70 

1.58 

0.0001 

C*S* SUBJ/G 

2662 

93.02 

TOTAL 


3455 


447.13 
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Experiment  2:  Novice  Subject  Comparability  Analysis  During 
the  Last  12  Training  Trials  (Control,  Frequency,  Sequence, 
and  Plan  Novices) 


Dependent 

Measure:  Total 

Operations 

Source 

df 

ss 

F-ratio 

GROUP  (G) 

3 

13.56 

0.22 

0.8850 

SUBJ/G 

44 

921.71 

TOTAL 

47 

935.27 

Dependent 

Measure:  Different  Operations 

Soui cc 

df 

SS 

F-ratio 

p-value 

GROUP  (G) 

3 

1.59 

0.78 

0.5101 

SUBJ/G 

44 

29.71 

TOTAL 

47 

31.30 

Dependent 

Measure:  Total 

Time 

Source 

df 

SS 

F-ratio 

GROUP  (G) 

3 

1335.15 

0.29 

0.8344 

SUBJ/G 

44 

68181.42 

TOTAL 


47 


69516.57 
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Dependent  Measure: 

Total 

Movement 

Source 

df 

ss 

F-ratio 

d- value 

GROUP  (G) 

3 

454.33 

0.04 

0.9883 

SUBJ/G 

44 

157799.32 

TOTAL 


47 


158253.65 
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Experiment  2:  Models  (Control  Novices,  Frequency  Novices 
Sequence  Novices,  Plan  Novices,  Experts)  by  Sessions 
(3  -  5) 


Dependent  Measure: 

Total 

Operations 

Source 

df 

SS 

F-ratio 

n-value 

Between  Subiects 

MODEL  (M) 

'4 

281.40 

2.76 

0.0367 

SUBJ/M 

55 

1403.46 

Within  Subiects 

SESSION  (S) 

2 

396.38 

35.38 

0.0001 

S*M 

8 

47.78 

1.07 

0.3922 

S* SUBJ/M 

110 

616.28 

TOTAL 

179 

2745.30 

Dependent  Measure: 

Different  Operations 

Source 

df 

SS 

F-ratio 

Between  Subiects 

MODEL  (M) 

4 

4.89 

1.27 

0.2919 

SUBJ/M 

55 

52.84 

Within  Subiects 

SESSION  (S) 

2 

6.07 

23.12 

0.0001 

S*M 

8 

0.59 

0.56 

0.8088 

S* SUBJ/M 

110 

14.43 

TOTAL 


179 


78.82 
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Dependent  Measure: 

Total 

Time 

Source 

d£ 

ss 

F-ratio 

D-value 

Between  Subiects 

MODEL  (M) 

4 

26288.13 

4.05 

0.0032 

SUBJ/M 

55 

80357.81 

Within  Subiects 

SESSION  (S) 

2 

48039.23 

72.79 

0.0001 

S*M 

8 

5319.48 

2.02 

0.0510 

S* SUBJ/M 

110 

36296.67 

TOTAL 

179 

196301.31 

Dependent  Measure: 

Total 

Movement 

Source 

df 

SS 

F-ratio 

D-value 

Between  Subiects 

MODEL  (M) 

4 

38906.94 

1.31 

0.2762 

SUBJ/M 

55 

407080.67 

Within  Subiects 

SESSION  (S) 

2 

67064.27 

23.96 

0.0001 

S*M 

8 

7920.88 

0.71 

0.6844 

S* SUBJ/M 

110 

153945.84 

TOTAL 


179 


674918.59 
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Experiment  2:  Models  (Slow  Novices  n  =  6,  Fast  Novices 
p  =  6,  Frequency  Novices  n  =  12,  Sequence  Novices  n  =  12, 


Plan  Novices  n  =  12 

,  Experts  n  =  12)  by 

Sessions 

(3  -  5) 

Dependent  Measure: 

Total 

Operations 

Source 

df 

Type  III  SS 

F-ratio 

Between  Subiects 

MODEL  (M) 

5 

597.45 

5.93 

0.0002 

SUBJ/M 

54 

1087.41 

Within  Subiects 

SESSION  (S) 

2 

367.57 

32.54 

0.0001 

S*M 

10 

54.14 

0.96 

0.4836 

S* SUBJ/M 

108 

609.91 

TOTAL 

179 

2745.30 

Dependent  Measure: 

Different  Operations 

Source 

df 

Tvoe  III  SS 

F-ratio 

Between  Subiects 

MODEL  (M) 

5 

13.48 

3.29 

0.0116 

SUBJ/M 

54 

44.25 

Within  Subiects 

SESSION  (S) 

2 

5.11 

19.25 

0.0001 

S*M 

10 

0.69 

0.52 

0.8752 

S* SUBJ/M 

108 

14.34 

TOTAL 


179 


78.82 
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Dependent  Measure: 

Total 

Time 

Source 

df 

Type  HI  SS 

F-ratio 

o-value 

Between  Subiects 

MODEL  (M) 

5 

45834.15 

8.14 

0.0001 

SUBJ/M 

54 

60811.79 

Within  Subiects 

SESSION  (S) 

2 

41753.47 

62.85 

0.0001 

S*M 

10 

5740.21 

1.73 

0.0834 

S* SUBJ/M 

108 

35875.94 

TOTAL 

179 

196301.31 

Dependent  Measure: 

Total 

Movement 

Source 

df 

Tvoe  III  SS 

F-ratio 

D-value 

Between  Subiects 

MODEL  (M) 

5 

111934.50 

3.62 

0.0069 

SUBJ/M 

54 

334053.11 

Within  Subiects 

SESSION  (S) 

2 

56765.91 

20.20 

0.0001 

S*M 

10 

10133.55 

0.72 

0.7029 

S* SUBJ/M 

108 

151733.17 

TOTAL 


179 


674918.59 
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Experiment  2:  Dialogue  (User,  Computer,  Mixed)  by 
Presentation  Order  (MUC,  MCU,  CMU,  CUM,  UCM  UMC)  by  Model 
(Control  Novices,  Frequency  Novices,  Sequence  Novices,  Plan 
Novices,  Experts) 


Dependent  Measure:  Total  Operations 


Source 

df 

SS 

F-ratio 

Between  Subiects 

MODEL  (M) 
PRESENTATION 

2 

63.82 

1.36 

0.2812 

ORDER  (0) 

5 

11.59 

0.10 

0.9911 

M*0 

10 

282.73 

1.21 

0.3491 

SUBJ/M*0 

18 

421.60 

Within  Subiects 

DIALOGUE 

INITIATIVE  (D) 

2 

50.50 

7.26 

0.0022 

D*M 

4 

11.38 

0.82 

0.5223 

D*0 

10 

431.00 

12.39 

0.0001 

D*M*0 

20 

100.81 

1.45 

0.1625 

D*  SUBJ/M*0 

36 

125.19 

TOTAL 


107 


1498.61 
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Dependent  Measure:  Different  Operations 


Source 

df 

Er ratio 

o  value 

Between  Subiects 

MODEL  (M) 

2 

1.03 

0.69 

0.5126 

PRESENTATION 

ORDER  (0) 

5 

5.66 

1.52 

0.2322 

M*0 

10 

7.11 

0.96 

0.5102 

SUBJ/M*0 

18 

13.38 

Within  Subiects 

DIALOGUE 

INITIATIVE  (D) 

2 

2.04 

10.28 

0.0003 

D*M 

4 

0.04 

0.09 

0.9852 

D*0 

1C 

7.27 

7.31 

0.0001 

D*M+0 

20 

2.11 

1.06 

0.4244 

D*SUBJ/K*0 

36 

3.57 

TOTAL 


107 


42.24 
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Dependent  Measure:  Total  Time 


Source 

df 

ss 

F-ratio 

D-value 

Between  Subiects 

MODEL  (M) 

2 

2098.06 

1.08 

0.3591 

PRESENTATION 

OnDER  (0) 

5 

11721  52 

2.42 

0.0757 

M  0 

10 

19019.24 

1.97 

0.1017 

SUBJ/M*0 

18 

17409.60 

Within  Subiects 

DIALOGUE 

INITIATIVE  (D) 

2 

3190.48 

5.03 

0.0119 

D*M 

4 

707.35 

0.56 

0.6949 

D*0 

10 

47856.27 

15.09 

0.0001 

D*M*0 

20 

4635.95 

0.73 

0.7696 

D*SUBJ/M*0 

26 

11420.11 

TOTAL 


107  118058.57 
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Dependent  Measure:  Total  Movement 


Source 

df 

ss 

F-ratio 

D-value 

Between  Subiects 

MODEL  (M) 

2 

16617.84 

1.10 

0.3538 

PRESENTATION 

ORDER  (0) 

5 

10139.55 

0.27 

0.9243 

M*0 

10 

76432.92 

1.01 

0.4695 

SUBJ/M*0 

18 

135808.12 

Within  Subiects 

DIALOGUE 

INITIATIVE  (D) 

2 

6696.48 

4.48 

0.0183 

D*M 

4 

3125.33 

1.05 

0.3922 

D*0 

10 

77655.75 

10.40 

0.0001 

D*M*0 

20 

21089.62 

1.41 

0.1798 

D*  SUB J/M*0 

36 

26890.51 

TOTAL 


107 


374456.13 
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Experiment  2:  Dialogue  Orders  (MUC,  MCU,  CMU,  CUM,  UCM, 
UMC,  Slow  Novices,  Fast  Novices,  Experts)  by  Sessions 
(3  -  5),  Experts  n  =  12,  Other  Groups  n  =  6 


Dependent  Measure: 

Total 

Operations 

Source 

df 

Type  HI  SS 

F-ratio 

p-value 

Between  Subiects 

DIALOGUE  ORDER  (0) 

8 

545.22 

3.05 

0.0070 

SUBJ/O 

51 

1139.65 

Within  Subiects 

SESSION  (S) 

2 

420.17 

45.77 

0.0001 

S*0 

16 

195.85 

2.67 

0.0015 

S* SUBJ/O 

102 

468.20 

TOTAL 

179 

2745.30 

Dependent  Measure: 

Different  Operations 

Source 

df 

Type  III  SS 

F-ratio 

p-value 

Between  Subiects 

DIALOGUE  ORDER  (0) 

8 

18.11 

2.91 

0.0093 

SUBJ/O 

51 

39.62 

Within  Subiects 

SESSION  (S) 

2 

6.20 

30.29 

0.0001 

S*0 

16 

4.59 

2.80 

0.0009 

S* SUBJ/O 

102 

10.44 

TOTAL 


107 


78.82 
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Dependent  Measure: 

Total 

Time 

Source 

df 

Type  HI  SS 

F-ratio 

D-value 

Between  Subiects 

DIALOGUE  ORDER  (0) 

8 

55457.61 

6.91 

0.0001 

SUBJ/O 

51 

51188.33 

Within  Subiects 

SESSION  (S) 

2 

50423.05 

89.96 

0.0001 

S*0 

16 

13031.29 

2.91 

0.0006 

S* SUBJ/O 

102 

28584.86 

TOTAL 

107 

196301.32 

Dependent  Measure: 

Total 

Movement 

Source 

df 

Tvne  III  SS 

F-ratio 

D-value 

Between  Subiects 

DIALOGUE  ORDER  (0) 

8 

105456.20 

1.97 

0.0688 

SUBJ/O 

51 

340531.41 

Within  Subiects 

SESSION  (S) 

2 

69697.73 

27.27 

0.0001 

s*o 

16 

31700.04 

1.55 

0.0961 

S* SUBJ/O 

102 

130166.68 

TOTAL 


107 


674918.59 
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Experiment  2:  User  interaction  with  the  Online  Aids  - 
Models  (Frequency,  Sequence,  Plan)  by  Sessions  (3  -  5) 

Dependent  Measure:  Advice/Trial 


Source 

df 

SS 

F-ratio 

r>-value 

Between  Subiects 

MODEL  (M) 

2 

6.40 

12.45 

0.0001 

SUBJ/M 

33 

8.47 

Within  Subiects 

SESSION  (S) 

2 

2.02 

4.43 

0.0156 

S*M 

4 

3.05 

3.34 

0.0150 

S* SUBJ/M 

66 

15.06 

TOTAL 

107 

35.00 

Dependent  Measure: 

Proportion 

of  Advised 

Trials 

Source 

d£ 

ss 

E-ratio 

D-value 

Between  Subiects 

MODEL  (M) 

2 

0.10 

0.69 

0.5083 

SUBJ/M 

33 

2.27 

Within  Subiects 

SESSION  (S) 

2 

0.30 

1.95 

0.1502 

S*M 

4 

0.22 

0.73 

0.5754 

S* SUBJ/M 

66 

5.08 

TOTAL 


107 


7.97 
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Experiment  2:  User  Interaction  with  the  Online  Aid  - 
Dialogue  Initiative  (User,  Computer,  Mixed)  by  Presentation 
Order  (MUC,  MCU,  CMU,  CUM,  UCM  UMC)  by  Model  (Frequency, 
Sequence,  Plan) 

Dependent  Measure:  Advice/Trial 


Source  df  SS  F-ratio  p-value 


Between  Subjects 


MODEL  (M) 
PRESENTATION 

2 

6.40 

10.48 

0.0010 

ORDER  (0) 

5 

1.00 

0.65 

0.6631 

M*0 

10 

1.99 

0.65 

0.7533 

SUBJ/M*0 

18 

5.49 

Within  Subjects 


DIALOGUE 


INITIATIVE  (D) 

2 

4.82 

15.08 

0.0001 

D*M 

4 

0.75 

1.17 

0.3389 

D*0 

10 

3.18 

1.99 

0.0646 

D*M*0 

20 

5.63 

1.76 

0.0686 

D*SUBJ/M*0 

36 

5.76 

TOTAL 


107 


35.00 
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Dependent  Measure:  Proportion  of  Advised  Trials 


Source 

df 

s§ 

F-ratio 

D-value 

OPOUP  (G) 

2 

0.10 

0.55 

0.5878 

PRESENTATION 

ORDER  (0) 

5 

0.34 

0.79 

0.5704 

G*0 

10 

0.36 

0.42 

0 . 9184 

SUBJ/G*0 

18 

1.56 

DIALOGUE 

INITIATIVE  (D) 

2 

3.30 

69.46 

0.0001 

D*G 

4 

0.27 

2.81 

0.0398 

D*0 

10 

0.57 

2.41 

0.0262 

D*G*0 

20 

0.62 

1.31 

0.2355 

D*SUBJ/G*0 

36 

0.85 

TOTAL 


107 


7.97 
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Experiment  2:  Transfer  Performance  for  Slow  Novices 
(n  =  6),  Fast  Novices  (n  =  6),  Frequency  Novices  (n  =  12), 
Sequence  Novices  (n  =  12),  Plan  Novices  (n  =  12),  and 
Experts  (n  =  12) 

Dependent  Measure:  Total  Operations 


Source 

df 

Type  III  SS 

F-ratio 

p-value 

GROUP  (G) 

5 

193.23 

4.84 

0.0011 

SUBJ/G 

54 

431.57 

TOTAL 

59 

624.79 

Dependent  Measure: 

Different  Operations 

Source 

df 

Tvoe  III  SS 

F-ratio 

GROUP  (G) 

5 

3.74 

3.05 

0.0169 

SUBJ/G 

54 

13.22 

TOTAL 

59 

16.96 

Dependent  Measure: 

Total 

Time 

Source 

df 

Tvpe  III  SS 

F-ratio 

GROUP  (G) 

5 

9209.88 

7.41 

0.0001 

SUBJ/G 

54 

13938.69 

TOTAL 


59 


23148.57 
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Dependent  Measure: 

Total 

Movement 

Source 

df 

Type  HI  SS 

F-r.a_tiQ 

Devalue 

GROUP  (G) 

5 

46202.41 

4.11 

0.0032 

SUBJ/G 

54 

121491.69 

TOTAL 

59 

167694.10 
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Experiment  2:  Search  Strategy  Analyses  for  Slow  Novices 
(n  =  6),  Fast  Novices  (n  =  6),  Frequency  Novices  (n  =  12), 
Sequence  Novices  (n  =  12),  Plan  Novices  (n  =  12),  Experts 
(n  =  12) 

Dependent  Measure:  Scroll  Up  Proportion 


Source 

df 

Tvt>e  III  SS 

Er ratio 

p-value 

GROUP  (G) 

5 

0.1934 

2.41 

0.0480 

SUBJ/G 

54 

0.8679 

TOTAL 

59 

1.0613 

Dependent  Measure: 

Scroll  Down  Proportion 

Source 

df 

Type  II I  SS 

F-ratio 

GROUP  (G) 

5 

0.4936 

2.44 

0.0455 

SUPJ/G 

54 

2.1856 

TOTAL 

59 

2.6793 

Dependent  Measure: 

Page 

Up  Proportion 

Source 

df 

Type  HI  SS 

F-ratio 

o-value 

GROUP  (G) 

5 

0.0178 

0.23 

0.9452 

SUBJ/G 

54 

0.8261 

TOTAL 


59 


0.8439 
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Dependent 

Measure:  Page 

Down  Proportion 

Source 

df 

Type  HI  SS 

F-ratio 

o-value 

GROUP  (G) 

5 

0.1501 

0.57 

0.7237 

SUBJ/G 

54 

2.8374 

TOTAL 

59 

2.9876 

Dependent 

Measure:  Section  Proportion 

Source 

df 

Tvoe  III  SS 

F-ratio 

p- value 

GROUP  (G) 

5 

0.2447 

2.13 

0.0746 

SUBJ/G 

54 

1.2386 

TOTAL 

59 

1.4833 

Dependent 

Measure:  Zoom 

In  Proportion 

Source 

df 

Type  III  SS 

F-ratio 

D-value 

GROUP  (G) 

5 

0.2105 

0.88 

0.5002 

SUBJ/G 

54 

3.3081 

TOTAL 

59 

3.3577 

Dependent 

Measure:  Zoom 

Out  Proportion 

Source 

df 

Type  HI  SS 

F-ratio 

p-Y.alu.s 

GROUP  (G) 

5 

0.1234 

5.10 

0.0007 

SUBJ/G 

54 

0.2616 

TOTAL 


59 


0.3851 


265 


Dependent  Measure:  File  Select  Proportion 


Source 

df  Type  HI  SS 

F-ratio 

D-value 

GROUP  (G) 
SUBJ/G 

5  0.2262 

54  1.8745 

1.30 

0.2757 

TOTAL 

59  2.1008 

Dependent 

Measure : 

Search  Proportion 

Source 

df  Tvoe  III  SS 

F-ratio 

GROUP  (G) 
SUBJ/G 

5  0.0572 

54  1.7904 

0.35 

0.8833 

TOTAL 

59  1.8477 

Dependent  Measure: 

Search-And  Proportion 

Source 

df  Tvoe  III  SS 

F-ratio 

p-yalue 

GROUP  (G) 
SUBJ/G 

5  0.0273 

54  0.4574 

0.65 

0.6683 

TOTAL 

59  0.4847 

Dependent 

Measure : 

Search-And-Not  Proportion 

Source 

df  Tvne  III  SS 

F-ratio 

D-value 

GROUP  (G) 
SUBJ/G 

5  0.0003 

54  0.0031 

1.02 

0.4172 

TOTAL 


59 


0.0034 
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Dependent  Measure: 

Index 

Proportion 

Source 

d£ 

Type  111  SS 

F-ratio 

D-value 

GROUP  (G) 

5 

0.0751 

1.52 

0.1993 

SUBJ/G 

54 

0.5351 

TOTAL 

59 

0.6101 
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Experiment  2:  Overall  Satisfaction  with  the  Online  Aids  - 
Models  (Frequency,  Sequence,  Plan)  by  Dialogue  Initiatives 
(User,  Computer,  Mixed) 

Dependent  Measure:  Unsatisfactory-Satisfactory 


Source 

d£ 

Type  HI  SS 

F-ratio 

D-value 

Between  Subiects 

MODEL  (M) 

2 

10.85 

1.87 

0.1703 

SUBJ/M 

33 

95.83 

Within  Subiects 

DIALOGUE 

INITIATIVE  (D) 

2 

0.57 

0.20 

0.8169 

D*M 

4 

0.95 

0.17 

0.9526 

D* SUBJ/M 

63 

88.17 

TOTAL 


104 


195.33 
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Experiment  2:  Reduced  Analysis  of  Coleman's  (1985) 
Individual  Bipolar  Scales  -  Scales  (2  -  9)  by  Model 
(Frequency,  Sequence,  Plan)  by  Dialogue  initiatives 
(User,  Computer,  Mixed) 

Dependent  Measure:  Mean  Rating  for  a  Dialogue  Presentation 
Order 


Source  df  SJ3  F-ratio  p-value 


Between  Subi ects 


MODEL  (M) 

2 

48.91 

1.93 

0.1797 

ORDER/M 

15 

190.16 

Within  Subi ects 

DIALOGUE 

INITIATIVE  (D) 

2 

3.91 

0.80 

0.4581 

D*M 

4 

15.83 

1.62 

0.1945 

D* ORDER/M 

30 

73.20 

SCALE  (S) 

7 

51.65 

7.85 

o.oooi 

S*M 

14 

24.62 

1.87 

0.0380 

S* ORDER/M 

105 

98.74 

D*  S 

14 

5.14 

0.81 

0.6548 

D*  S*M 

28 

8.81 

0.70 

0.8728 

D*S*ORDER/M 

210 

94.94 

TOTAL 


104 


615.92 
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Experiment  2:  Individual  Bipolar  Scales  and  Online  Aids  - 
Models  (Frequency,  Sequence,  Plan)  by  Dialogue  Initiatives 
(User,  Computer,  Mixed) 

Dependent  Measure:  Unnatural-Natural 


Source 

df 

Type  HI  SS 

F-ratio 

o-value 

MODEL  (M) 

2 

11.54 

1.91 

0.1641 

CTTBJ/M 

33 

99.67 

TOTAL 

104 

195.33 

Dependent 

Measure:  Undependable-Dependable 

Source 

df 

Tvne  III  SS 

F-ratio 

D-value 

MODEL  (M) 

2 

32.25 

4.79 

0.0149 

SUBJ/M 

33 

111.12 

TOTAL 

104 

204.19 

Dependent 

Measure:  Complex-Simple 

Source 

df 

Type  III  SS 

F-ratio 

MODEL  (M) 

2 

9.33 

1.84 

0.1755 

SUBJ/M 

33 

83.84 

TOTAL 


104 


187.43 


270 


Dependent 

Measure : 

Slow- 

Fast 

Source 

df 

Type  HI  SS 

F-ratio 

p-value 

MODEL  (M) 

2 

21.61 

3.17 

0.0552 

SUBJ/M 

33 

112.62 

TOTAL 

104 

212.23 

Dependent 

Measure : 

Incomplete-Complete 

Source 

df 

Type  III  SS 

F-ratio 

o-value 

MODEL  (M) 

2 

7.95 

0.86 

0.4329 

SUBJ/M 

33 

152.67 

TOTAL 

104 

236.53 

Dependent 

Measure: 

Disgusting-Pleasing 

Source 

df 

Type  HI  SS 

F-ratio 

MODEL  (M) 

2 

25.90 

3.37 

0.0465 

SUBJ/M 

33 

126.74 

TOTAL 

104 

237.56 

Dependent 

Measure : 

Uncooperative-Cooperative 

Source 

df 

Type  III  SS 

F-ratio 

D-value 

MODEL  (M) 

2 

21.55 

2.53 

0.0948 

SUBJ/M 

33 

140.36 

TOTAL 


104 


247.20 
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Dependent  Measure: 

Useless 

-Useful 

Source 

df 

Type  HI  SS 

F-ratio 

•D-value 

MODEL  (M) 

SUBJ/M 

2 

33 

18.39 

121.27 

2.50 

0.0973 

TOTAL 

104 

221.43 
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Experiment  2:  Reduced  Analysis  of  Last  3  Individual 
Bipolar  Scales  Used  to  Assess  the  Advice-Giving 
Characteristics  of  the  Online  Aids  -  Scale  (10  -  12)  by 
Model  (Frequency,  Sequence,  Plan)  by  Dialogue  Initiatives 
(User,  Computer,  Mixed) 

Dependent  Measure:  Mean  Rating  from  a  Dialogue  Presentation 
Order 


Source 

df 

SS 

F-ratio 

D-value 

Between  Subiects 

MODEL  (M) 

2 

16.44 

1.49 

0.2574 

ORDER/M 

15 

82.90 

Within  Subiects 

DIALOGUE 

INITIATIVE  (D) 

2 

1.40 

0.42 

0.6623 

D*G 

4 

2.13 

0.32 

0.8635 

D*ORDER/M 

30 

50.19 

SCALE  (S) 

7 

1.84 

0.47 

0.6283 

S*M 

14 

1.85 

0.24 

0.9150 

S* ORDER/M 

105 

58.53 

D*  S 

14 

1.26 

0.77 

0.5467 

D*S*M 

28 

6.27 

1.93 

0.0725 

D*S*ORDER/M 

210 

24.42 

TOTAL 


104 


247.24 


Appendix  K 

FREQUENCY  PROFILES  OF  EXPERT  SUBJECTS 


Search  Procedures 


Search 


Cluster 

1 

2 

3 

£ 

1 

6 

1 

8 

9 

IQ 

11 

12 

1  | 

21 

72 

13 

28 

5 

177 

44 

9 

1 

21 

7 

188 

2  ! 

24 

49 

21 

24 

3 

118 

41 

0 

3 

68 

3 

133 

3  1 

40 

17 

31 

19 

5 

86 

37 

0 

12 

48 

43 

89 

4  1 

14 

29 

12 

18 

12 

107 

6 

0 

16 

56 

51 

75 

5  1 

9 

14 

4 

12 

5 

26 

0 

0 

4 

11 

36 

33 

6  1 

25 

26 

12 

21 

5 

43 

3 

0 

10 

28 

30 

30 

7  1 

28 

55 

19 

29 

17 

70 

27 

0 

18 

20 

39 

110 

8  | 

37 

54 

12 

21 

15 

61 

24 

3 

17 

61 

21 

63 

9  1 

28 

30 

12 

10 

18 

39 

16 

0 

47 

23 

29 

63 

10  | 

30 

40 

8 

16 

21 

36 

16 

0 

49 

26 

28 

63 

11  1 

7 

9 

7 

7 

10 

18 

1 

0 

22 

3 

16 

14 

12  1 

18 

25 

5 

10 

14 

25 

5 

0 

12 

4 

20 

17 

Search  Procedures: 

1.  SCROLL  UP 

2.  SCROLL  DOWN 

3.  PAGE  UP 

4.  PAGE  DOWN 


5.  SECTION 

6.  SEARCH 

7.  SEARCH-AND 

8.  SEARCH-AND-NOT 


9.  INDEX 

10.  ZOOM  IN 

11.  ZOOM  OUT 

12.  FILE  SELECT 


Cluster  descriptions  are  given  on  the  next  page. 
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Cluster 

Cluster 

Cluster 

Cluster 

Cluster 

Cluster 

Cluster 

Cluster 

Cluster 

Cluster 

Cluster 

Cluster 


1:  Specific  APC  Inventory. 

2:  Specific  Tank  Inventory. 

3:  Unspecified  APC  Inventory  (Embedded). 

4:  Unspecified  Tank  Inventory  (Embedded). 

5:  Unspecified  APC  Inventory  (Nonembedded). 
6:  Unspecified  Tank  Inventory  (Nonembedded). 
7:  Equipment  for  APC. 

8:  Equipment  for  Tank. 

9:  Combat  Support  Information  (Subgroups). 
10:  Combat  Support  Information  (Mission  and 
Operations) . 

11:  Army  Operations  (Numerical). 

12:  Army  Operations  (Verbal). 


Appendix  L 


TRANSITION  MATRICES  AND  INITIAL  PROFILES  OF 
EXPERT  SUBJECTS 


Transition  Matrix  and  Initial  Profile  for  Cluster  1 
Specific  APC  Inventory 


To 

Initial 

Search 

From  1 

2 

3 

4 

5 

6 

7 

s 

2 

10 

11 

12 

Profile 

1|  15 

17 

1 

1 

0 

2 

0 

0 

0 

1 

1 

0 

0 

2 1  13 

42 

2 

1 

1 

4 

0 

0 

0 

2 

2 

0 

2 

3  |  1 

2 

3 

7 

2 

0 

1 

0 

0 

2 

2 

0 

0 

4 1  1 

4 

7 

10 

1 

8 

2 

0 

0 

10 

0 

2 

11 

5 1  0 

0 

0 

2 

2 

2 

0 

0 

0 

1 

2 

1 

1 

6 1  8 

50 

5 

9 

0 

24 

4 

0 

0 

13 

3 

0 

4 

7 1  2 

6 

1 

0 

0 

13 

44 

37 

0 

0 

0 

0 

0 

8 1  0 

0 

0 

0 

0 

0 

2 

16 

0 

0 

0 

0 

0 

9  |  0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

10 1  0 

0 

0 

7 

2 

14 

0 

0 

0 

4 

4 

0 

1 

11 1  1 

1 

0 

1 

2 

1 

0 

0 

0 

3 

0 

4 

0 

12  |  0 

2 

0 

6 

1 

114 

64 

0 

1 

2 

0 

5 

185 

Search  Procedures 

: 

1.  SCROLL  UP 

5. 

SECTION 

9 

.  INDEX 

2.  SCROLL  DOWN 

6. 

SEARCH 

10 

.  ZOOM 

IN 

3.  PAGE 

UP 

7. 

SEARCH 

-AND 

11 

.  ZOOM 

OUT 

4.  PAGE 

DOWN 

8. 

SEARCH 

-AND- 

■NOT 

12 

.  FILE 

SELECT 
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Transition  Matrix  and  Initial  Profile  for  Cluster  2 
Specific  Tank  Inventory 


Ifl 


From  123456282101111 


Initial 

Search 

Profile 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 


10  9 

12  28 
2  1 
1  2 
0  0 
8  27 

1  3 

0  0 
0  0 
0  5 

0  0 
0  10 


0  0 
2  3 

5  5 

13  10 

0  0 

6  8 

1  1 
0  0 
0  0 
0  6 
0  0 
0  7 


0  6 
0  3 
0  0 
0  4 
0  0 
0  21 
0  3 
0  0 
0  0 
3  29 
0  1 
0  78 


0  0 

1  o 
0  0 
1  0 
0  0 

2  0 

41  0 

0  0 
0  0 
2  0 
0  0 
37  0 


0  1 
0  0 
0  2 
0  1 
0  3 
0  17 
0  0 
0  0 
1  0 
0  13 
0  0 
2  0 


0  0 
0  0 
0  0 
0  0 
0  0 
1  0 
0  0 
0  0 
1  1 
1  33 
0  2 
0  1 


0 

0 

0 

1 

0 

0 

0 

0 

1 

67 

0 

99 


Search  Procedures: 


3 .  SCROLL  UP 

2.  SCROLL  DOWN 

3.  PAGE  UP 

4.  PAGE  DOWN 


5.  SECTION 

6.  SEARCH 

7.  SEARCH-AND 

8.  SEARCH-AND-NOT 


9.  INDEX 

10.  ZOOM  IN 

11.  ZOOM  OUT 

12.  FILE  SELECT 
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Transition  Matrix  and  Initial  Profile  for  Cluster  3 
Unspecified  APC  Inventory:  Embedded 


£o 

Initial 

Search 

From  1 

2 

3 

4 

5 

6 

1 

S 

9 

10 

11 

12  Profile 

1|  31 

3 

0 

1 

0 

1 

0 

0 

0 

24 

2 

14  0 

2  |  7 

12 

0 

0 

0 

6 

1 

0 

1 

0 

1 

0  1 

3 1  2 

1 

7 

7 

0 

2 

0 

0 

0 

12 

4 

8  0 

41  1 

0 

12 

8 

0 

3 

1 

0 

1 

7 

8 

1  2 

5 1  0 

0 

0 

0 

2 

0 

0 

0 

3 

2 

0 

0  0 

6 1  19 

5 

12 

6 

0 

9 

0 

0 

0 

4 

35 

0  2 

7 1  16 

3 

8 

0 

0 

2 

37 

0 

0 

1 

4 

0  1 

8 1  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

9  |  0 

0 

5 

0 

1 

0 

0 

0 

1 

0 

2 

7  1 

10  J  0 

3 

0 

6 

0 

35 

10 

0 

1 

7 

2 

1  16 

11|  0 

2 

1 

7 

1 

5 

3 

0 

2 

2 

15 

36  0 

12  |  0 

3 

0 

4 

3 

62 

30 

0 

8 

8 

2 

6  73 

Search  Procedures 

: 

1.  SCROLL  UP 

5. 

SECTION 

9 

.  INDEX 

2.  SCROLL  DOWN 

6. 

SEARCH 

10 

.  ZOOM  IN 

3.  PAGE 

UP 

7. 

SEARCH- 

-AND 

11 

.  ZOOM  OUT 

4.  PAGE 

DOWN 

8. 

SEARCH- 

-AND- 

-NOT 

12 

.  FILE  SELECT 
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Transition  Matrix  and  Initial  Profile  for  Cluster  4 
Unspecified  Tank  Inventory:  Embedded 


From 

1 

2 

3 

£ 

2 

To 

6 

2 

8 

2 

10 

II 

12 

Initial 

Search 

Profile 

1| 

10 

7 

2 

1 

0 

0 

1 

0 

0 

5 

2 

2 

0 

2  1 

7 

17 

0 

1 

0 

5 

1 

0 

0 

0 

2 

2 

0 

3  1 

0 

0 

1 

2 

0 

1 

0 

0 

2 

6 

1 

1 

0 

4 1 

0 

1 

6 

9 

0 

7 

1 

0 

1 

4 

0 

1 

0 

5  | 

0 

0 

0 

0 

2 

2 

0 

0 

1 

4 

0 

6 

1 

6  1 

7 

22 

3 

5 

0 

4 

0 

0 

0 

14 

45 

10 

1 

7  1 

2 

0 

1 

0 

0 

1 

6 

1 

0 

0 

1 

0 

0 

8  | 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

9  i 

1 

0 

0 

1 

3 

0 

0 

0 

1 

5 

0 

9 

4 

10  | 

0 

4 

0 

5 

4 

44 

4 

0 

4 

24 

3 

4 

48 

11  1 

1 

1 

2 

7 

1 

3 

1 

0 

2 

7 

2 

35 

0 

12  | 

0 

4 

0 

2 

5 

72 

1 

0 

5 

1 

3 

7 

54 

Search  Procedures: 


1. 

SCROLL  UP 

5. 

SECTION 

9. 

INDEX 

2. 

SCROLL  DOWN 

6. 

SEARCH 

10. 

ZOOM  IN 

3. 

PAGE  UP 

7. 

SEARCH-AND 

11. 

ZOOM  OUT 

4. 

PAGE  DOWN 

8. 

SEARCH-AND-NOT 

12. 

FILE  SELECT 

279 


Transition  Matrix  and  Initial  Profile  for  Cluster  5 
Unspecified  APC  Inventory:  Nonembedded 


To 

Initial 

Search 

From  1 

2 

3 

4 

5 

£ 

7 

a 

2 

10 

11 

22 

Profile 

1  |  7 

5 

1 

0 

1 

1 

0 

0 

1 

0 

1 

1 

0 

2  |  6 

12 

0 

0 

1 

2 

0 

0 

1 

0 

11 

0 

0 

3 1  0 

0 

0 

1 

0 

0 

0 

0 

0 

2 

1 

0 

0 

4 1  3 

2 

2 

2 

0 

0 

0 

0 

1 

3 

9 

0 

2 

5 1  1 

1 

0 

1 

2 

0 

0 

0 

0 

1 

2 

2 

0 

6 1  5 

1 

0 

0 

0 

3 

2 

0 

1 

2 

20 

1 

2 

7 1  0 

1 

1 

0 

0 

0 

3 

1 

0 

0 

2 

1 

0 

8 1  0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

1 

0 

0 

9 1  0 

1 

0 

1 

1 

0 

0 

0 

0 

0 

2 

2 

0 

10 1  0 

5 

0 

4 

0 

2 

0 

0 

0 

3 

1 

0 

0 

11  1  0 

1 

0 

3 

2 

3 

0 

0 

1 

1 

1 

32 

0 

12  |  0 

8 

0 

7 

2 

23 

3 

0 

3 

5 

0 

7 

32 

Search  Procedures 

1.  SCROLL  UP 

5 

.  SECTION 

9 

.  INDEX 

2.  SCROLL  DOWN 

6 

.  SEARCH 

10 

.  ZOOM 

IN 

3.  PAGE 

UP 

7 

.  SEARCH- 

-AND 

11 

.  ZOOM 

OUT 

4.  PAGE 

DOWN 

8 

.  SEARCH- 

-AND- 

-NOT 

12 

.  FILE 
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Transition  Matrix  and  Initial  Profile  for  Cluster  6 
Unspecified  Tank  Inventory:  Nonembedded 
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Transition  Matrix  and  Initial  Profile  for  Cluster  7 
Equipment  for  APC 


To 
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Transition  Matrix  and  Initial  Profile  for  Cluster  8 
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Transition  Matrix  and  Initial  Profile  for  Cluster  9 
Combat  Support  Information:  Subgroups 
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Transition  Matrix  and  Initial  Profile  for  Cluster  10 
Combat  Support  Information:  Mission  and  Operations 


To  Initial 
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Transition  Matrix  and  Initial  Profile  for  Cluster  11 
Army  Operations:  Numerical 
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Transition  Matrix  and  Initial  Profile  for  Cluster  12 
Army  Operations:  Verbal 
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Appendix  M 

SEARCH  PLANS  OF  EXPERT  SUBJECTS 


Final  Search  Plan  for  Cluster  1 
Specific  APC  Inventory 


STEPS 

SUGGESTED  SEARCH  PLAN 

1. 

FILE  SELECT 

(to  an  APC  inventory  file) 

2. 

SEARCH,  SEARCH  AND,  or  SEARCH  AND  NOT 
(for  specific  inventory  information) 

3. 

SCROLL  UP  or  SCROLL  DOWN  (optional) 

(to  verify  or  compare  inventory 
information) 

Final  Search  Plan  for  Cluster  2 

Specific  Tank  Inventory 

STEPS 

SUGGESTED  SEARCH  PLAN 

1. 

FILE  SELECT 

(to  a  tank  inventory  file) 

2. 

SEARCH  or  SEARCH  AND 

(for  specific  inventory  information) 

3. 

SCROLL  UP  or  SCROLL  DOWN  (optional) 

(to  verify  or  compare  inventory 
information) 
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Final  Search  Plan  for  Cluster  3 
Unspecified  APC  Inventory:  Embedded 


STEPS  SUGGESTED  SEARCH  PLAN 

1.  FILE  SELECT 

(to  a  APC  inventory  file) 

2 .  SEARCH 

(for  specific  inventory  information) 

3.  If  target  is  not  located  ZOOM  OUT 
(and  repeat  steps  1  and  2) 


Final  Search  Plan  for  Cluster  4 
Unspecified  Tank  Inventory:  Embedded 


STEPS  SUGGESTED  SEARCH  PLAN 

1.  FILE  SELECT 

(to  a  tank  inventory  file) 

2 .  SEARCH 

(for  specific  inventory  information) 


3. 


If  target  is  not  located  ZOOM  OUT 
(and  repeat  steps  1  and  2) 
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Final  Search  Plan  for  Cluster  5 
Unspecified  APC  Inventory:  Nonembedded 


STEPS 

SUGGESTED  SEARCH  PLAN 

1. 

FILE  SELECT 

(to  an  APC  inventory  file) 

2. 

SCROLL  DOWN 

(for  APC  availability  information) 

3. 

If  target  is  not  located  ZOOM  OUT 
(and  repeat  steps  1  and  2) 

Final  Search  Plan  for  Cluster  6 

Unspecified  Tank  Inventory:  Nonembedded 

STEPS 

SUGGESTED  SEARCH  PLAN 

1. 

FILE  SELECT 

(to  an  tank  inventory  file) 

2. 

SCROLL  DOWN 

(for  tank  availability  information) 

3. 

If  target  is  not  located  ZOOM  OUT 
(and  repeat  steps  1  and  2) 
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Final  Search  Plan  for  Cluster  7 
Equipment  for  APC 


STEPS 

SUGGESTED  SEARCH  PLAN 

1. 

FILE  SELECT 

(to  the  motorized  rifle  equipment  file) 

2. 

SEARCH  or  SEARCH  AND 

(for  motorized  rifle  statistics) 

3. 

SCROLL  UP  or  SCROLL  DOWN  (optional) 

(to  verify  or  compare  equipment 
statistics) 

Final  Search  Plan  for  Cluster  8 
Equipment  for  Tank 


STEPS 

SUGGESTED  SEARCH  PLAN 

1. 

ZOOM  IN  or  FILE  SELECT 

(to  the  tank  equipment  file) 

2. 

SEARCH  or  SEARCH  AND 
(for  tank  statistics) 

3. 

SCROLL  UP  or  SCROLL  DOWN  (optional) 

(to  verify  or  compare  equipment 
stati sties ) 
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Final  Search  Plan  for  Cluster  9 
Combat  Support  Information:  Subgroups 


Steos 

DescriDtion 

1. 

FILE  SELECT 

(to  a  combat  support  file) 

2. 

INDEX 

(to  locate  subgroup  information) 

3. 

If  target  is  not  located  ZOOM  OUT 
(and  repeat  steps  1  and  2) 

Final  Search  Plan  for  Cluster  10 

Combat  Support  Information:  Mission  and  Operations 

STEPS 

SUGGESTED  SEARCH  PLAN 

1. 

FILE  SELECT 

(to  a  combat  support  file) 

2. 

INDEX 

(to  locate  information  on  mission  or 
operational  capabilities) 

3. 

If  target  is  not  located  ZOOM  OUT 
(and  repeat  steps  1  and  2) 
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Final  Search  Plan  for  Cluster  11 
Army  Operations:  Numerical 


STEPS 

SUGGESTED  SEARCH  PLAN 

1. 

Position  to  the  army  operations  file 
(if  necessary  ZOOM  OUT) 

2. 

INDEX  or  SEARCH 

(to  locate  personnel  information) 

Final  Search  Plan  for  Cluster  12 

Army  Operations:  Verbal 

STEPS 

SUGGESTED  SEARCH  PLAN 

1. 

Position  to  the  army  operations  file 
(if  necessary  ZOOM  OUT) 

2. 

SEARCH 

(to  locate  specific  verbal  information) 

3. 


SCROLL  UP  or  SCROLL  DOWN  (optional) 
(to  verify  verbal  information) 


Appendix  N 

INSTRUCTIONS  FOR  THE  COMMAND-SELECTION  MODELS 


ONLINE  ASSISTANCE  INSTRUCTIONS 
(Frequency-Based  Advice) 

In  the  experimental  trials  to  follow,  you  will  have  an 
online  assistant  to  help  in  the  selection  of  search 
commands.  This  online  assistant  was  developed  from  the 
performance  of  experienced  users  on  this  information 
retrieval  system.  Consequently,  the  command-selection 
advice  offered  by  the  assistant  could  be  useful  for  finding 
information . 

The  information  contained  in  this  online  assistant 
consists  of  the  frequency  of  search  procedure  selection  by 
experienced  users.  The  online  assistant  will  monitor  your 
use  of  commands  and  will  be  able  to  give  advice  on  search 
procedures  that  were  used  by  experienced  users. 

Advice  will  be  given  by  highlighting  three  search 
procedures  on  the  touch  keypad.  These  highlighted  keys  are 
the  three  search  procedures  most  frequently  selected  by 
experienced  users.  In  addition,  a  ranking  will  also  be 
provided  below  each  highlighted  search  procedure  indicating 
the  relative  frequency  of  use  by  experienced  users  (i.e.,  1 
-  most  frequently  used,  2  -  second  most  frequently  used,  and 
3  -  third  most  frequently  used). 

As  an  example,  Figure  1  displays  command-selection  advice 
that  might  be  given  to  a  subject.  (Please  note  that  Figure 
1  is  not  presented,  but  a  similar  example  is  given  in  the 
body  of  the  dissertation  as  Figure  9.)  The  online  assistant 
has  highlighted  and  ranked  the  advised  search  procedures  in 
terms  of  selection  frequency  by  experienced  users.  In  this 
example,  FILE  SELECT  is  the  most  frequently  selected  search 
procedure  followed  by  SEARCH  and  ZOOM-IN.  Consequently, 
with  this  advice  the  subject  might  decide  to  use  the  one  of 
these  three  search  procedures  next. 

The  online  assistant  will  give  command-selection  advice 
through  three  different  dialogues.  In  the  first  dialogue. 
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the  online  assistant  will  give  advice  when  you  make  a 
request  by  pressing  a  HELP  key.  In  the  second  dialogue,  the 
online  assistant  will  decide  when  to  give  advice.  Finally, 
in  the  third  dialogue,  you  will  be  able  to  request  advice 
and  the  computer  will  also  offer  advice  if  necessary.  Only 
one  type  of  dialogue  with  the  assistant  will  be  available 
during  each  of  the  three  next  sessions. 

ONLINE  ASSISTANCE  INSTRUCTIONS 
(Sequence-based  Advice) 

In  the  experimental  trials  to  follow,  you  will  have  an 
online  assistant  to  help  in  the  selection  of  search 
commands.  This  online  assistant  was  developed  from  the 
performance  of  experienced  users  on  this  information 
retrieval  system.  Consequently,  the  command- sequence  advice 
offered  by  the  assistant  could  be  useful  for  finding 
information. 

The  information  contained  in  this  online  assistant 
consists  of  the  initial  selection  frequency  of  search 
-.mmands  and  of  the  sequence  of  search  procedures  used  by 
experienced  users.  For  example,  the  online  assistant  knows 
how  frequently  a  FILE  SELECT  is  used  as  the  initial  search 
procedure  and  how  frequently  SEARCH  is  used  after  the  FILE 
SELECT  procedure  is  selected.  Therefore,  the  online 
assistant  will  monitor  your  use  of  commands  and  will  be  able 
to  give  advice  on  what  search  procedures  to  use  considering 
the  search  procedure  selected  previously. 

Advice  will  be  given  by  highlighting  three  search 
procedures  on  the  touch  keypad.  These  highlighted  keys  will 
be  the  three  search  procedures  most  frequently  selected  by 
experienced  users  either  initially  or  considering  a  previous 
search  command  (depending  on  whether  advice  is  given  at  the 
start  or  in  the  middle  of  a  trial).  In  addition,  a  ranking 
will  also  be  provided  below  each  highlighted  search 
procedure  indicating  the  relative  frequency  of  use  by 
experienced  users.  (i.e  ,  1  -  most  frequently  used,  2  - 
second  most  frequently  used,  and  3  -  third  most  frequently 
used) . 

As  an  example,  Figure  1  displays  command- sequence  advice 
that  might  be  given  to  a  subject  after  the  selection  of  the 
FILE  SELECT  procedure.  (Please  note  that  Figure  1  is  not 
presented,  but  a  similar  example  is  given  as  Figure  10  in 
the  body  of  the  dissertation.)  The  online  assistant  has 
highlighted  and  ranked  the  advised  search  procedures  in 
terms  of  frequency  of  selection  by  experienced  users.  In 
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this  example,  SEARCH  would  be  the  most  frequently  selected 
search  procedure  after  the  selection  of  the  FILE  SELECT 
procedure,  followed  by  SEARCH  AND  and  SCROLL  DOWN. 
Consequently,  with  this  advice  the  subject  might  decide  to 
use  one  of  these  three  search  procedures  next. 

The  online  assistant  will  give  command- sequence  advice 
through  three  different  dialogues.  In  the  first  dialogue, 
the  online  assistant  will  give  advice  when  you  make  a 
request  by  pressing  a  HELP  key.  In  the  second  dialogue,  the 
online  assistant  will  decide  when  to  give  advice.  Finally, 
in  the  third  dialogue,  you  will  be  able  to  request  advice 
and  the  computer  will  also  offer  advice  if  necessary.  Only 
one  type  of  dialogue  with  the  assistant  will  be  available 
during  each  of  the  three  next  sessions. 

ONLINE  ASSISTANCE  INSTRUCTIONS 
(Plan-based  Advice) 

In  the  experimental  trials  to  follow,  you  will  have  an 
online  assistant  to  help  in  the  selection  of  search 
commands.  This  online  assistant  was  developed  from  the 
behavior  of  experienced  users  on  this  information  retrieval 
system.  Consequently,  the  command-selection  advice  offered 
by  the  assistant  could  be  useful  for  finding  information. 

The  information  contained  in  this  online  assistant 
consists  of  the  sequence  of  search  procedure  use  by 
experienced  users,  as  well  as  some  information  on  target 
characteristics.  For  example,  the  online  aid  knows  the  most 
frequently  used  sequence  of  search  procedures  for  a  class  of 
targets.  Therefore,  the  online  assistant  will  monitor  your 
use  of  commands  and  will  be  able  to  give  advice  on  what 
seerch  procedures  to  use.  In  addition,  it  will  provide 
information  on  what  target  characteristics  to  look  for 
during  information  retrieval.  Together,  the  sequence  of 
search  procedures  and  the  target  characteristics  define  what 
will  be  called  a  search  plan. 

Advice  will  be  given  by  displaying  the  search  plan  in  the 
work  area  of  the  primary  display  and  highlighting  the 
appropriate  search  procedures  on  the  touch  keypad  (see 
Figure  1).  (Please  note  that  Figure  1  is  not  presented,  but 
a  similar  example  is  given  in  the  body  of  the  dissertation 
as  Figure  11.)  The  search  plan  may  be  two  or  three  steps 
long  and  may  contain  one,  two,  or  three  alternative  search 
p.ocedures  in  each  step.  The  numbers  below  each  of  the 
highlighted  search  procedures  on  the  touch  keypad  correspond 
to  the  step  in  the  search  plan. 
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As  an  example.  Figure  1  displays  a  search  plan  for 
finding  a  target.  This  search  plan  directs  the  subject  to 
use  the  FILE  SELECT  procedure  to  position  to  a  tank 
inventory  file,  followed  by  a  SEARCH  or  SEARCH  AND  procedure 
to  find  specific  inventory  information.  Finally,  an 
optional  SCROLL  UP  or  SCROLL  DOWN  step  is  listed  to  verify 
or  compare  inventory  information. 

The  online  assistant  will  give  plan-based  advice  through 
three  different  dialogues.  In  the  first  dialogue,  the 
online  assistant  will  give  advice  only  when  you  make  a 
request  by  pressing  a  HELP  key.  In  the  second  dialogue,  the 
online  assistant  will  decide  when  to  give  advice.  Finally, 
in  the  third  dialogue,  you  will  be  able  to  request  advice 
and  the  computer  will  also  offer  advice  if  necessary.  Only 
one  type  of  dialogue  with  the  assistant  will  be  available 
during  each  of  the  three  next  sessions. 


Appendix  0 

INSTRUCTIONS  FOR  THE  DIALOGUE  INITIATIVES 


COMPUTER- INITIATED  ADVICE 

In  this  dialogue  with  the  online  assistant  only  the 
computer  will  determine  when  to  give  advice  on  the  selection 
of  search  commands.  That  is,  the  online  assistant  will 
monitor  your  selection  of  search  commands  and  give  advice  if 
it  determines  that  you  are  using  search  procedures  unlike 
experienced  users. 

When  giving  advice,  the  online  assistant  will  highlight 
and  rank  the  search  procedures  most  frequently  used  by 
experienced  users.  The  online  assistant  will  only  give 
advice  once  during  a  trial,  and  once  presented,  the  advice 
will  remain  on  the  display.  In  addition,  it  is  important  to 
note  that  the  online  assistant  will  cancel  the  search 
procedure  you  selected  when  providing  advice.  You  can, 
however,  select  the  cancelled  search  procedure  again  after 
the  advice  is  given. 

We  would  like  to  encourage  you  to  try  to  use  the 
command-selection  information  given  by  online  assistant 
since  it  will  probably  be  useful  for  assisting  and  improving 
your  information  retrieval  skills.  However,  please  remember 
that  just  like  a  human  assistant,  the  online  assistant  may 
sometimes  provide  inappropriate  advice. 

If  you  have  any  questions  at  this  time,  please  ask  the 
experimenter . 


USER- INITIATED  ADVICE 

In  this  dialogue  with  the  online  assistant  you  will  have 
a  HELP  key  to  request  command-selection  advice.  The  HELP 
key  is  the  large  key  on  the  right  side  of  the  keyboard. 

In  response  to  your  pressing  the  HELP  key,  the  assistant 
will  highlight  and  rank  the  search  procedures  most 
frequently  used  by  experienced  users.  You  can  request 
advice  at  any  time  during  the  trial,  however,  once  advice  is 
presented  it  will  remain  for  the  rest  of  the  trial. 
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We  would  like  to  encourage  you  to  try  to  use 
command- selection  information  available  through  the  online 
assistant  since  it  will  probably  be  useful  for  assisting  and 
improving  your  information  retrieval  skills.  However, 
please  remember  that  just  like  a  human  assistant,  the  online 
assistant  may  sometimes  provide  inappropriate  advice. 

If  you  have  any  questions  at  this  time,  please  ask  the 
experimenter. 

COMPUTER-  AND  USER- INITIATED  ADVICE 

In  this  dialogue  with  the  online  assistant  you  will  have 
a  HELP  key  to  request  command- selection  advice.  The  HELP 
key  is  the  large  key  on  the  right  side  of  the  keyboard.  You 
can  request  advice  at  any  time  during  the  trial.  In 
addition,  the  online  assistant  will  also  be  monitoring  your 
selection  of  search  commands  and  will  indicate  when  it  has 
advice  available  by  displaying  the  highlighted  message  "If 
you  want  advice  press  HELP"  directly  below  the  touch  keypad. 

In  response  to  your  pressing  the  HELP  key,  the  assistant 
will  highlight  and  rank  the  search  procedures  most 
frequently  used  by  experienced  users.  This  advice  will  then 
remain  on  the  display  for  the  rest  of  the  trial. 

We  would  like  to  encourage  you  to  try  to  use 
command-selection  information  available  through  the  online 
assistant  since  it  will  probably  be  useful  for  assisting  and 
^...proving  your  information  retrieval  skills.  However, 
please  remember  that  just  like  a  human  assistant,  the  online 
assistant  may  sometimes  provide  inappropriate  advice. 

If  you  have  any  questions  at  this  time,  please  ask  the 
experimenter . 


Appendix  P 


INSTRUCTIONS  AND  PRACTICE  FOR  THE  BIPOLAR 
ADJECTIVE  SCALES 


INSTRUCTIONS 

At  this  time  we  would  like  you  to  evaluate  the  file 
search  assistance  that  you  received  during  this  session.  To 
evaluate  the  file  search  assistance,  12  scales  with  two 
adjectives  will  be  presented,  one  scale  at  a  time.  The 
first  scale  you  will  receive  will  be  an  overall  evaluation 
of  the  file  search  assistance.  '  This  overall  scale  will  be 
followed  by  11  individual  scales  for  evaluating  the 
assistance . 

Your  task  will  be  to  rate  the  assistance  using  the  two 
adjectives  on  a  seven-point  scale.  To  demonstrate,  an 
example  will  follow  asking  you  to  evaluate  your  last 
vacation. 


(All  subjects  would  then  receive  the  overall  satisfaction 
scale  and  two  individual  bipolar  scales.  Therefore,  the 
three  bipolar  adjectives  that  subjects  would  practice  by 
evaluating  their  last  vacation  were: 

1.  UNSAT IS FACTORY- SATISFACTORY 

2.  HECTIC-RESTFUL 

3.  HEALTHY-UNHEALTHY.) 
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Appendix  Q 

FORM  FOR  COLLECTING  THE  DIALOGUE  PREFERENCE 

RANKINGS 

Please  rank  order  the  online  assistance  dialogues  from  1  to 
3. 

1  -  signifies  your  most  preferred  dialogue  and  3  -  signifies 
your  least  preferred  dialogue. 


Computer- Initiated 
User-Initiated 
Mixed- Initiated 


i 
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VITA 


JAY  ELKERTON 


Birth  Date: 
Marital  Status: 


EDUCATION 


Graduate:  Virginia  Polytechnic  Institute  and  State 

University 
Blacksburg,  Virginia 

Masters  and  Ph.  D.  candidate.  Industrial 
Engineering  and  Operations  Research 
(Human  Factors) 


Undergraduate:  University  of  Minnesota 
Minneapolis,  Minnesota 
B.A.  summa  cum  laude.  Psychology,  1981 


PROFESSIONAL  EXPERIENCE 

10-85  to  12-85:  Instructor  in  Engineering  Psychology, 
Department  of  Industrial  Engineering  and  Operations 
Research,  VPI  &  SU.  Responsible  for  development  of 
course  materials  and  lectures  in  a 
senior- level/first-year  graduate  course  in  human 
factors. 

9-84  to  9-85:  Graduate  Research  Assistant  in  the  Human 

Factors  Laboratory,  Department  of  Industrial 
Engineering  and  Operations  Research,  VPI  &  SU. 
Responsible  for  the  development  and  evaluation  of 
adaptive  software  interfaces  on  a  contract  sponsored 
by  the  U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences. 
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6-84  to  9-84:  Graduate  Research  Assistant  in  the  Human 

Factors  Laboratory,  Department  of  Industrial 
Engineering  and  Operations  Research,  VPI  &  SU. 
Responsible  for  a  critical  review  of  adaptive 
human-computer  software  interfaces  on  a  contract 
sponsored  by  the  U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences,  Systems  Research 
Laboratory. 

1-84  to  6-84:  Graduate  Teaching  Assistant  for  the 

Department  of  Industrial  Engineering  and  Operations 
Research,  VPI  &  SU.  Partially  responsible  for  the 
development  of  class  materials  and  activities  in  a 
course  devoted  to  human  performance  in  industrial 
systems . 

9-81  to  12-83:  Graduate  Research  Assistant  in  the  Human 
Factors  Laboratory,  Department  of  Industrial 
Engineering  and  Operations  Research,  VPI  &  SU. 
Responsible  for  the  development  of  experimental 
software  and  the  design  of  experiments  for  a  project 
on  human- computer  interaction  and  decision  behavior 
which  was  sponsored  by  the  Office  of  Naval  Research. 

5-79  to  8-81:  Undergraduate  Research  Assistant  in  the 
Department  of  Psychology,  University  of  Minnesota. 
Assisted  in  the  psychomotor  assessment  of  identical 
twins  raised  apart  and  the  assessment  of  an  individual 
with  extraordinary  memory  skills.  Conducted  research 
on  human  memory  with  prompted  recall  culminating  in  a 
Summa  thesis. 
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